.Net Foundation On .NET, 16 Jan 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Understanding the Event Processor Host in Azure Event Hubs

Description

Azure Event Hubs Documentation
https://docs.microsoft.com/en-us/azure/event-hubs?WT.mc_id=ondotnet-c9-cxa

In this episode, Serkant Karaca and Shubha Vijayasarathy from the Azure Event Hubs team join us to talk about how and when to use the Event Processor Host for Azure Event Hubs in our .NET applications. They'll discuss use cases and cover topics like partitioning, and also show how to use the .NET SDK for Event Hubs.

[00:30] - Processing data in Event Hubs
[02:38] - How the event processor host creates receivers
[06:24] - Checkpointing in the event processor host
[08:11] - Scaling out event processing
[11:13] - Balancing the number of partitions and event processor host
[13:18] - Recommendations for consuming messages
[14:12] - Azure Functions and the event processor host

GitHub repo with samples
https://github.com/Azure/azure-event-hubs/tree/master/samples/DotNet/Microsoft.Azure.EventHubs?WT.mc_id=ondotnet-c9-cxa

Enable reliable messaging for Big Data applications using Azure Event Hubs
https://docs.microsoft.com/en-us/learn/modules/enable-reliable-messaging-for-big-data-apps-using-event-hubs?WT.mc_id=ondotnet-c9-cxa

Choose between Azure messaging services
https://docs.microsoft.com/en-us/azure/event-grid/compare-messaging-services?WT.mc_id=ondotnet-c9-cxa

Azure IoT Documentation
https://docs.microsoft.com/en-us/azure/iot-hub?WT.mc_id=ondotnet-c9-cxa

Send events to or receive events from Azure Event Hubs using .NET Core
https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-dotnet-standard-getstarted-send?WT.mc_id=ondotnet-c9-cxa

Microsoft.Azure.EventHubs NuGet package
https://www.nuget.org/packages/Microsoft.Azure.EventHubs/

A

In this episode of the on that note, show we're gonna, learn about the event processor hose for Azure event, hubs check it out.

A

Everyone welcome to another episode of the on dotnet show and I have the folks from the azure event up Stephen joining me again. We're gonna talk a little bit more about event, processing with a dirty event hubs. So in our last episode we talked about how event hubs specifically is used for very hype throughput. You know multiple consumers, multiple producers.

A

So what do we talk a little bit more in detail about how exactly we process that information and how the clients send them? Yes,.

B

Absolutely so, when you're talking about the receiving sent or the consuming part of event hubs, you have data coming into this big buffer, which is even perhaps, and then you have data being distributed in partitions, and then you are thinking about how do I consume this now we briefly touched upon you know: okay, you have the event receivers you're consuming from all these partitions in the event hubs.

B

So what happens or what we have seen is when the data is very small, it's fine, but when you're you're thinking of even taps and you're thinking of scale consuming parallely among these very large scale becomes very challenging because you are thinking of load, balancing you're thinking of orchestrating your consumers.

B

So what we have built is a simple sdk, simple Orchestrator called the even processor host and that helps you with your downstream processing in parallel and certain tests, probably going to dig deep into what components of even processor is I mean what does even processor host what components does it have, and how do you take advantage of this simple Orchestrator? Okay,.

A

Sure so, let's do a circle. What do.

C

We have seen how you create an event implied. On top of that, you create the receivers right.

B

C

Receiver is now dedicated to receive from a single partition. Okay, so when it comes to, you know large number of partition scenario. So how do you you launcher like receivers and then start managing them? So in order for this issue, we have a wrapper around the low-level API, which is the receiver itself. So basically you fire up an amount of process rolls and then this process rolls will go and then start grating the receivers and start consuming the messages from those receivers.

A

And so on, we, when we create this processor house, can we can we determine or can we dictate how many receivers as a creator? Does it kind of just create them based on some criteria, so.

C

That's what we'll do that's the first thing it will go and then discover how many partitions are available for that, given you on top and then it will create single receiver for each partition.

B

C

Then the temptation itself is pretty much like straightforward. We are provided an event processor which will probably implementing their own message. Senders in the process. Event. I think all we hand over the all the messages that are received from the particular particular receivers.

C

B

Even processor, who's kind of hides how it's creating or how it's talking to the low-level receivers. This is something that the customers need not worry about, so it's just a complete wrapper, so what the customers using even processor host would be worrying about, is how do I scale up my processor, our consumers? How do I checkpoint and what are the things that I need a checkpoint and how? How does even processor host help me with load balancing? My consumers, like if I want to add more if I want to remove, will happen.

B

How does the consume help so so.

A

In the previous video, we saw how to use like the client to actually say: hey, like you know, receive this message, and we did that.

B

In the loop, exactly essentially.

A

What this is doing is creating its imagining an event loop for us. Essentially, you.

B

Know you can kind of consider, but there are lots more things. It's.

C

Happening yeah background as.

B

Well, yeah, okay,.

C

It's pretty much, this one is single procedure, so when you fire it up, then it will just create all these years. It will own all the resource, yeah.

A

C

It becomes more like a better solution when you start running multiple cores, so.

B

C

Let's say in order to log balance- or you know, through it, you're processing loads to multiple CPUs, multiple VMs and you can just start host as many as possible and they will start sharing the partitions. That.

B

C

For example, I will give an example: you are receiving from say, 12 partitioning on top and then, if you start 4 processor, on 4, different VMs, so each VM alone or become responsible for only processing 3 of the.

A

C

A

It so could this be a situation where I have something like darker, kubernetes and I might have an events host processor right. That's.

C

A

Potter poor container right now, I spread these across. Like that's.

B

Great help you, if you add in more nodes, you can just scale up your consumer so that you're increasing your parallel processing and the idea is you're. Always most customers want. You know, want a stream in near real time, so they want low, latency fast processing. So that's what this kind of helps got.

C

It ok, so I think we can continue this sample.

C

Once you create the event processor horse with by providing the a host name, which needs to be dedicated, so each host, no wild checkpointing, what type of what partitions they're owning in an event top name where they must be consumed from consumer group name. And then the connection string name, connect string, as we know, and on top of that you need to provide a source connection string and the list content name which will be used for check pointing in these management. Ok, so.

B

Even processor host has this check pointing mechanism where it's just check pointing and leading into a storage into a separate component which, for this I mean for by default, we used as a storage, blob storage blobs to define the leases and then checkpoint and load balance among the consumers. We do have option where and if anybody wants to bring in their own store, they can have an interface if you have an interface like an implement and then use their own store for checkpointing. So the the storage component is only for checkpointing and leasing data.

A

So that solves for me something very similar to what the folks in turbo functions. Yeah I know they have a check pointing mechanism.

B

Probably little bit difference of what you're talking about.

A

But they also use blob storage, Chris.

B

Yeah, that's right, it's mainly for leasing into finding it's easy. Even you are just taking your that load from your Orchestrator and maintaining it in a different store. Okay,.

C

Okay, all right, if you switch back to our sample so.

A

C

You create your first rows. Basically, you should you notice start running it, you register it and then register call will start and then discover the partitions. And then your hosts will start grabbing. Parties are created, receivers on top of them and then start assuming you.

B

Basically, you're trying to register a host into the server saying hey: this is my listener. This is where my data from even thoughts will be coming into so.

A

B

Of a thing that's happening in the background, gotcha.

A

A

Love when we were recording videos and visual studio starts to spin.

C

Okay, so in this one it's going very quick.

A

So you're only the event host right now.

B

Yes, processor host and it's trying to receive from all the partitions as if I'm, seeing it right, yeah.

C

Since we are, we were already running this Center for a while.

B

C

Now trying to grab every message available and then process them right.

A

Like we were saying earlier, if you wanted to scale this up or scale this out a little bit.

B

A

I could just spin up some additional nodes. The same yeah.

B

I'd like to show that yeah, probably just you know, bring up another even processor host and then another instance of the processor host. And you can see how the least just gets grabbed as.

C

You can see on the left side our senator still running on.

B

The right side.

C

Now our process process is running the same pace. Every time we send a message. It is.

B

C

In one of the partitions, and then our process rolls is graphing it. It's.

B

Almost like there is no latency there and.

C

What if we spawn another instance of process arose I'm gonna, just use.

B

C

Another one any one so once I start a new process rows, so it registers itself and then, as you can see, it starts grabbing. One of the partitions.

B

So basically it's doing some.

C

Sort of dancing.

B

C

So this the first time I ran as a single processor Rose, its own doll for available partitions and.

B

You can see that that partition in the previous window. You can see that partition one is being grabbed by the other host here, as.

C

You can see yeah second horse once we start it. First grabbed partition three and I started racing from partition tree and then, after some time it decided no, it's not there. Yet it means to grab another one.

B

C

Make it even so each horse will own two partitions.

B

And yeah it grabbed.

C

Partition two as well yeah.

B

C

The new horse is grabbing the partitions, it's stealing the partitions, as you can see on the first sauce, it says these loss, it lost the partition, so it stopped receiving from that partition. So once they reach a balanced state, as you can see, the first horse week spawn is owning partition, zero and one, as you can see it's receiving from zero.

B

C

The second one is receiving from 23, so.

B

If you add one more host, it'll start balancing among three and.

C

Let's say another scenario is: what happens if one of those crashes writes or stops processing, so I say we stop this son and then you can expect the leases to expire and the this singles. The running alone now finds.

B

C

One is receiving from this partition, so I should start a partition, pump or start receiving from this part.

A

C

A little time, depending on your configuration for the process, yeah.

A

C

Default, our lease duration is 30 seconds, so it takes two seconds for the lease to expire. Okay,.

A

So one of the things I was wondering about when you're showing it is. Is there like an optimal balance between the amount of partitions? You have the amount of post processors you're at like so, for instance, we saw that there were three partitions right and when you split up an extra holes like one holes had like one and two and then the other.

B

Man, 3 yeah, so.

A

What if I, had like three partitions and like I had five? Yes.

B

I would step back a little bit and say the gowning factor with event. Hops is to put unit or the amount of bandwidth at here reserved. So if you have resolved one throughput unit, what it entitles you is 1,001 KB messages are 1 MB per second of ingress and 2 MB or 2001 Kb messages per second of egress. So if you were to use even processor host to consume from your event, hops, no matter how many partitions you have no matter how many hosts you have the max that you can egress.

B

If you have one throughput is 2. It's 2, MB, 2 MB per second 1 through so as I as a golden rule of thumb. What we say is Mark your partitions are decide on your partitions, based on a number of throughput units that you would want to reserve or buy and based on your partitions, you can decide how many consumers or how many house hosts how many? How do you, how fast and how quickly you want to scale up and use your consumers, so it kind of influences from throughput all the wait'll.

B

Okay, your processors yeah that make sense so.

C

As you can see, after stopping the first horse now, the second horse is owning all the partitions, so this is receiving from all four parties available. Ok,.

A

Cool looks interesting, so is there anything else that you'd like to tell us about the event host processor, I, think.

C

A

So and a lot of folks are probably gonna, be looking at this video and wondering well, where can I get this code and where can I see the samples, so we got to make sure that your pulse video, yes.

B

We're gonna have.

A

These on github, somewhere or.

B

A

Public repository all attitude, absolutely.

B

Yes, Cooper yeah. Well, there's one thing that we always recommend is when you're consuming. We always ask our customers to use the eph way, or they even processor host way of consuming, rather than the direct receivers yeah.

A

B

Have direct receivers when you're thinking of building some kind of a connector like you know, you want double connector between your spark streaming and your even talks like you have data in your event. Hubs and you have a spark cluster and I want to kind of a connector to kind.

A

B

Some kind of a connector, some kind of a heavy body implementation that you are building your low-level receivers, help a lot, otherwise, all the load, balancing all the checkpointing that you need to take care of eph kind of offers you that so that is, that is like a best practice or recommendation from the product group that we have.

A

Actually another you're saying that another scenario kind of pops into my head, so do we have like, as your functions bindings for these things? Yes can I have my yeah.

B

A

Your functions consume this thing.

B

I could push that on somewhere. That's right and I should functions. Also, in the background use this even processor host the the binding as your function, binding from even helps to measure functions, is kind of using the eph as well. That's.

A

Great, so, if I wanted to do this in a server this way, mm-hmm- and you know- I- don't necessarily have to write that event. Processor, host, :.

B

Exactly yes, it's.

A

Kind of just already built into the platform so.

B

Yeah, absolutely yes, that's right! Cool.

A

B

A

Thank you all so much man. This is. This is a lot of great information, and thank you all for watching. This is another episode of two on that next show we learned about how we can use the event host processor for Azure event, hubs in dotnet.

A