Calyptia Fluentcon Europe 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Demystify the Fluent Core Event Loop and Join in on its Optimization Efforts - Matthew Fala, Amazon

Description

Demystify the Fluent Core Event Loop and Join in on its Optimization Efforts - Matthew Fala, Amazon

At the most fundamental level, Fluent Bit is an event driven system that relies on a core event loop running on each thread to initiate, resume, and clean routines required for delivering data. Have the core Fluent Bit event loop demystified, and deep dive into the data uncovering existing problems that occur due to the event loop in high throughput use-cases. The talk brings you into the community discussion on how to best optimize this event loop system and uncovers efforts spent on implementing the identified improvements. For the sake of understandably, a highly correlated analogy will be conveyed to modal the central concepts (the event loop, coroutines, cpu, yield, ready list, blocking wait) providing means to think through event loop problems and ideate solutions conceptually without full knowledge of unrelated systems. Attendees should after the talk be able to: 1. Understand the Event Loop system conceptually 2. Gain insight into the conceptual problems of the Event Loop system 3. Gain insight into the currently proposed conceptual solutions 4. Have the knowledge and tools to join the discussion, and propose new solutions.

A

Hello, my name is matthew falla and I will be um our presenter today on discussing the flipbit event, loop systems and also some of the optimization efforts that we took recently. uh So just a little bit about myself. um Yeah I've been working on this pr that just got approved and merged into flip at 1.9.

A

It changes up a little bit of the event loop system transitioning over from a fifo event, loop to a priority queue that we'll get into what exactly that is in just a little bit um and so yeah, so we're uh we just wanted to share um a little bit about kind of our findings, uh the improvement, uh metrics and statistics, and also kind of what you can do about the flip community and what aws can do to join in on the optimization efforts.

A

um So a little bit about myself, I work for aws, amazon, web services and also I am on the ecs observability team.

A

So we work heavily with flint bit and on flip it integrations trying to build sort of images that are on top of fluid bit to help our customers send logs easily to amazon services and also wherever else they would like to send their logs or metrics, and also a little bit about me is that I studied at the university of southern california, where I got my bachelor's in computer science um and yeah, I'm just thrilled to be presenting to you kind of our findings on the uh flip bit event loop today, uh so yeah, the the title of this presentation is uh flip bit event: loop, uh demystify it and join in on optimization efforts.

A

So I hope that after this talk, uh we can all be kind of up to speed on what the event loop is uh and also you know what optimization efforts are sort of being taken um to improve it and also what can be taken in the future, um all right so yeah. Let's begin with, I think the best way, which is the customer impact. uh Why should we care about the event loop in the first place and optimizing it so uh yeah?

A

Let's just start with statistics, um so our customers are kind of sharing with us that we're they're getting like 50 broken pipe errors and 49 connection timeout errors within a span of five minutes and they're, seeing this just kind of appear in their flip it logs- and this is not very good so, um each time you see one of these errors that triggers a retry and that it means that there's a genuine network error, that's occurring, and this is quite a bit. It's like 100 errors in five minutes which uh to degrade performance.

A

uh When does this happen, so um you can see that in low throughput cases uh we're seeing it's very, relatively stable as soon as you start sending logs at like 40 megabytes per second plus um for some, maybe maybe some of the bigger companies- or uh you know, for uh you know, like maybe uh one kilobyte logs 40 40 megabytes per second.

A

So uh in those cases, you'll be starting to see the errors on the right hand, side so connection, timed outs, broken pipes and connection initialization errors, uh so we kind of took to the whiteboard trying to figure out some ideas of what could be causing these problems uh at first, we thought maybe it's api issues, so maybe you were thinking that the place that we're sending our logs are having network problems like fire hose, but then we we realized that you know other customers are using fire hose without flip fit and they're not seeing these problems.

A

So we talked that out. uh The second is that it could be like a timeout problem, so maybe fire hose is working. It's just we're not waiting long enough, but it clearly says that um it's taking 10 seconds plus and yeah, so we said no there's no way that fire hose could genuinely be taking uh 10 seconds for the network request to complete and sure enough. We we verified that yeah fire host does not take. um You know 10 seconds to complete.

A

uh In fact, what we did when we profiled zip is we found that the network calls complete within around 100, milliseconds or less. However, when you have some code that makes a network call and flip bit, it actually takes around. You know 10 seconds to complete the line of code. That's that's responsible for making the network call. So what we'll see that the network call itself the actual network activity completes in about 100 seconds, but the code when it gets to that point takes you know, 10 seconds to to move to the next line.

A

So, what's going on there's a little bit of hanging whenever you make a network called movement, we found out that there's some some hangs. uh You know it hangs for like 10 seconds and at least these problems appear right afterwards. um So this is just a small diagram of what we found is happening. When some code is running, uh the code gets suspended to make a network call just kind of wait for the network call.

A

The network call completes really quickly, but then it takes over 10 seconds for the code to get back into the running state and that's the problem. So just a um some metrics that we were collecting uh yeah. So you can see the blue and the silver. The blue represents broken pipe errors. You see they're very heavily clustered and then the silver is the connection. Timeout. Also there's a lot of clusters. You can see uh maybe more clusters uh and um yeah, but but uh but yeah.

A

This is just a lot of errors that are occurring so in order to really get to the bottom of what what the issues are and how we can resolve it. We needed to really look into how fluidbit manages code right, how fluid manages suspending code and resuming code, um and so to do that we have to get to the very base layer uh and that's the event loop. So that's how flip it handles running code and so um yeah. That's what this presentation is gonna, be about.

A

It's gonna be about demystifying the event loop, so that we could resolve these problems and then joining in on the optimization efforts, and we hope that we can guide you through that, so that you can also join in as well um so kind of a quick summary of the things that we want to cover um event driven programming, which is kind of the fundamental of how fluid bit paradigm sort of works in terms of events and code.

A

Then next is a thought framework to think about the event loop that we've been using here at aws and then second is third, is the event, leave improvement and then next is going to be a network stability improvement chart so just swipes and metrics and then priority event loop problems. So we might have a chance to take a look at some of the problems with the current solution and then some future work that we can. We can all sort of work on uh to further optimize the pet loop, all right great.

A

So, let's begin with the event programming paradigm. So this is a paradigm that flip it really uses where you can make sort of blocking network calls without hogging the cpu.

A

So if you're looking at some programming language like java or python, when you make like a network call or when you do some sort of blocking operation, what you'll usually see is that the code or the thread will sort of block and and not allow anything else to run for a certain amount of time when you make a network call, so that's kind of what we're seeing right here.

A

It's a synchronous, um networking operation, so the code just blocks waits for the network, call to complete, and then the code will resume um and in that time maybe the cpu. The thread will not be doing anything. It'll just be kind of like sleeping waiting for the event and flip it. However, we see something a little bit smarter, so this is closer to like maybe like a javascript node.js style, um networking right where what happens is when you run some code and the code stops on a network call what hap.

A

Actually, some other work begins to run on the cpu, so the cpu is not being idle and once the network call completes, uh the code gets triggered to to get resumed um and yeah when it's when it's able to resume they'll just start running again so um yeah. This is very, uh very good, very beneficial for uh heavy io operation code. uh So when there's a lot of network calls taking place, uh you're not just going to let the cpu be idle for every time you make a network call all right, so uh yeah.

A

Let's take a look at how this is exactly working, so the code is running for a little bit. uh The network call gets made when the network call finishes. This triangle thing appears right, so that's your event and the event triggers the code to run so yeah. That's. This is kind of like how your standard network call will be made.

A

So key concept of event-driven programming, so this is maybe a little bit more of a complicated scenario that shows these events. um Is that there's always these events and the events trigger the code to run. So whenever you have code that you need to run, you always make an event. First, the event gets triggered and the trigger defense starts the code. So here you can see that we have two events.

A

We have the start code event and start other work event so same as in the previous slide great, so the start code event gets triggered. Both of these events are triggered at first, only one of them gets to be processed, so it starts the code.

A

The code hits a network call, so it gets suspended um and it's waiting on that network call to complete the other work event now flip bit, uh realizes that you know, there's no work uh being done because the code's been suspended, and so it decides. Okay, let's go look through all of our other events. Is there anything else? That's that's. You know triggered and it finds this other work event. This is great.

A

Okay, um this other work event tells us to start up the other work, so it goes ahead and runs the other work now, while that other work is running, um the the network call finishes, and so that generates a new event called the network event.

A

That doesn't really mean too much for fluidpay at this point, because it's just processing the other work until the other work completes, so other work completes and all of a sudden flip, it says: oh no work is being done. Let's take a look at our events and in this case it sees oh there's a network event great. uh What does this tell us to do? Well, it tells us to resume the code, so it resumes the code and that's it. That's our more advanced scenario.

A

It's a more advanced example of event loop, so you can see that we ran some code and the key concept is that all the code is sort of being triggered to run by these events. And you know, if you want to run some new code, just make an event, let it get triggered and then, when it gets triggered, then the code will run and a really key part of this is that all this is running on just one single thread, and so it's kind of a awesome.

A

um You know paradigm uh for for managing asynchronous operations, just all on one single thread without having to manage multi-threading or anything complicated like that and not allowing for the cpu to be idle so yeah all. This is why fluentfit is extremely efficient and you can take a look online for other documents on venture programming if you'd like to learn more okay, so event driven programming.

A

What is the event loop, so this is kind of the next section, so um diving in a bit uh the event loop is kind of like this layer that everything else all the code and fluid bit runs on top of. um So you kind of think about it as like this fundamental base layer that schedules everything, but that runs that sort of system that we talked about where events trigger code to run.

A

Let's take a look at the different responsibilities in fluid because flip it has an event loop and also the code that sits on top. So the code's responsibility is to code, essentially all fluid bits, so all of the plugins, the core fluent bit everything is sorted in this code section.

A

The code also is responsible for triggering and to not uh not just triggering but sorry registering, events to the event loop and also removing those events from the event loop once they get triggered the event loop. However, its job is to monitor those events, so maybe, like a network call event and when those events get triggered, like the network all completes, it needs to run the code, um that's associated with that event, so you can see there's kind of like a bi-directional connection. The code puts events.

A

The event loop puts events into code, all right um great, so this sort of bridge that's connecting these two sections is the events themselves.

A

So the events bridge together code, pointers and also event file, descriptors, so file descriptors, are just kind of like a way that the operating system can monitor different files um or maybe not files, because in linux everything is file. So you can monitor things like network calls, timers timeouts anything really that the operating system needs to take a look at on windows. This might be something sort of similar on like sockets, but in flint bit we just have some kind of abstraction and call of these things file descriptors.

A

These events are added to the event loop, all right so for the code, the only thing we really have to care about, because there's different types of code just to be aware of that. There's, like callbacks uh core code, that's embedded into the event loop and code genes. We really care about optimizing the code in case, so that's mainly what we're going to be taking a look at in uh in in the next few minutes yep. So the curtains include all the plug-in code.

A

It includes sort of input and output, filter, filter code, and that's mainly what we need to optimize.

A

So there's a bit of an issue with discussing um how to improve the event loop and that's that things can get really complicated very quickly because you're talking about code, routines yields, connection, timeouts, ready lists and a lot of different things that are sort of working together to create the system that everything else can be built on top of. So at aws.

A

We found that the best way to think about the event leave is with an analogy, and this allows us to really uh really just sort of scratch: our spitball ideas towards improving the event loop without having to dive into the implementation and the nitty-gritty details, but looking at it from a high level that allows us to conceptually target, you know the best solutions possible and really optimize the from a conceptual perspective and then allowing for those implementation details to get worked out whenever we decide to choose a different path to optimize a little bit of that loop.

A

So, let's take a look at this analogy.

A

So the analogy is the following and we're about to explain it. So this is it but we're going to go through it, one by one, there's some people at uh desk and they're filling out these forms and they're making calls to their friends when they need answers for the form so going through it one by one. uh There's these people um they're they're, just kind of like waiting around and these people represent the cover teams. uh Then there's a desk.

A

um This desk represents the cpu, so the person needs to be at the desk to do some work and then there's forms so forms kind of represent the code. When you're at the desk. You can be filling out your form, you can be, uh you know, adding answers to the forms, but what, if you don't know an answer to the form? Well, then you have to step away from the desk. Call your friend um get the answer.

A

So that's kind of this bone-a-friend section there in the bottom, all right: where are they going to be doing this well they're, going to be doing this in the phone booth, so the phone booth represents a not ready list. So stepping away from the desk represents and calling your friend represents, doing a network call. This phone booth represents the not ready list so um there's just kind of a list of these coachings.

A

These people that are making network calls and they're not waiting in the desk line and and lastly, that desk line represents a first in first out, evently ready list. So when you finish making your network call when you finish calling your friend and you have an answer for your form. Well, you can't write it down immediately because you don't have the desk, you don't have the cpu.

A

So what you need to do is stand in line for the desk and once you get to the very front of the line, then you will have access to your code, which is the form, and you can write down the answer and continue on your work, filling out other questions on the forum, all right so um yeah, going through an example, we have one person, who's filling out his form he's at the very front of the line uh he gets to a question that requires him to phone a friend.

A

So that's like a network call or something so he steps into the phone of friend line away from the desk. um The code is still around it's just no longer being processed by a car routine.

A

At this point, um someone else uh comes steps up to the desk who's waiting in line the next person and he grabs his code, his form and begins filling it out um and when he, when he gets a network, caller finishes then he'll, step away from the desk um and then next someone who is phoning their friend gets an answer from their friend. So that's like the network call completes and what they do is they need to stand back in line great. So that's the whole example.

A

So yeah flipbitland, um we already kind of covered it, but uh essentially all keyboard code, routines they're waiting in line for the desk to to get to the cpu uh want to make a network call. They have to step away and then, when the network all completes, then they can step back in line and then wait for the desk to write down their answer and continue on their work, all right, great, um so obstacles. There are a couple of different things that we have to keep in mind.

A

The first is that these cover teams have a problem remembering their answer once they get it from their friend. So when they stand back in line, if the line takes over ten minutes, uh then over two minutes, then they will forget their answer, so in flipbit land. This is kind of like saying that, if you finish your network call and then you are starting to wait to go, write down the answer to process that network call to complete it.

A

If you wait too long, there will be a broken type, pipe or connection timeout error, which is not a good thing.

A

So so what we need to do is make sure that this line can be short enough, that you can remember your answer when you get to the desk after you phone your friend. uh The second constrainer obstacle is that person switching takes time.

A

So when you stand in when you change from just a line to a desk and a new person takes a desk that takes a bit of time, so we don't want to add more person switches. What we have right now is fine.

A

Okay, so there's a problem with the first in first out um event loop, which we're looking at right here, so you can see the first and first step, because when you stand in line, then you have to kind of wait in line uh for all the people in front of you to get to the desk to get to the cpu in order to begin running your code again and here's the problem once this desk line gets really long.

A

All of a sudden, when you finish phoning your friend, you have to stand in this really long line. That's maybe like 10 minutes, and when you get to your desk, you forget your your. um Your network calls result so there's a broken pipe or connection timeout problem.

A

So where are these people coming from? How can this line get so long?

A

Well, the answer is that uh all these people are are kind of like inactive proteins right so they're these carotenes are sort of queued by the input plugins. So in flip it you have like these input, plugins that are generating the data they're, generating all the logs to get sort of flushed by the output plugins. Well, with the old fluid bits so flip it 1.8.

A

What would happen is that you would just have all of these sort of data, all these sort of proteins waiting uh to be sent over um to the desk line right, so they're waiting to go from inactive to active and, unfortunately, um the old solution which was in flip 1.8, is to take all these people immediately when you see them uh every time.

A

There's a flush move all of the people who are inactive, who just got generated by the input plugins to active so to the desk line, and what that what that causes is is for the desk line to get really long.

A

So you can imagine that if you are um creating like a hundred different input, uh chunks of data um or a hundred different code routines on input, plugins immediately those hundred proteins get turned into active proteins. Then that's a lot of protein to be processing at the same time and that's what we're seeing we're seeing uh sometimes around like 100 or more currents being processed at the exact same time and you'll you'll see that because there's so many currencies being processed. At the same time.

A

This line gets extremely long and uh people forget their answers by the time they get them to the desk. So uh what how we can resolve this is with a new policy.

A

So the new policy is that we need to have the people in hold line, so the input plugins uh sort of output we need to have them wait um until everyone at the desk has cleared. So what that does. Is it make sure that the desk line is as short as possible?

A

So if there's someone kind of like in this first first row um who's waiting at the desk line, then we will not allow anyone to be admitted from the hold line into the desk line.

A

However, if you look at the next section, if the desk line is empty, so that might mean that there's still people processing network calls at the phone booth, but if the desk line is empty, then we can start admitting people from the whole line.

A

To do this, we need to implement a priority queue so um yeah just going to the new policy. This is same slide yeah, so we can see that that now we're keeping the uh desk line extremely short, so you can see that that people get to the desk. They make a network call when the network call completes, they go back to the desk line and we're not allowing anyone else to join this sort of group of coaches until um all the people on the desk are clear.

A

So we can keep this this desk line very short and to implement that we need we. We can just make really there be two priorities, so the inactive care teams have a lower priority to hit the desk than the people who are waiting at the desk, so the active care routines. Those funds can get a higher party and get prioritized, and just by doing that, we can implement this policy.

A

So how does this look if we revisit the event loop? Well, um if you remember before we had these events, they were all having the same priority before now, we're going to add priorities. On top of that, so you can see that we added numbers to these events um and added some priorities, so the people so kind of looking at on the right hand, side. On the left hand, side, um the people are representing currencies so kind of remembering to our event driven programming.

A

These events trigger code or co routines to run, so each of these events have kind of people attached to them. So now we can prioritize the people right, so the people who are inactive, we give them an event, that's linked to them with a priority of uh two right, so they don't get the priority. uh And then the people who are active right, who are making network calls who are already standing in the desk line.

A

They get a priority event of one, and so these events, when they get triggered uh we'll, always make sure to prioritize um the higher priority, and you can see that when you do choose an event, one event gets triggered. Well, then, you send one of these people to the active coaching spot at the desk right, which is this top section above the event loop, which is writing the code.

A

All right, so, let's take a look at the change of responsibilities, so um the code responsibilities everything's the same except the code also needs to set a appropriate priority to the events. However, this is not mandatory, because um this was implemented such that you don't have to choose a priority. um If you don't choose it, then it'll just be the default party, uh which is what we had previously and everything will work the same and then next is the event loop.

A

So the event loop everything's the same, except we're going to make sure that if, if a couple events are triggered, a couple events are active: we're going to process through the higher priority events first and lastly, for the events everything's the same, except um instead of just having code, pointers and um file descriptors that we're monitoring for the events. What we need to also add is the priority, which is optional great.

A

Let's take a look at the improvements now that we implemented this, so this pr was implemented and merged into 1.9, um so yeah we're doing a lot of testing on 1.8.11 uh and we're also doing it under a bunch of different slowdowns. So we were adding some delays a little bit and see how they would react, uh how the network uh problems would get impacted by slower machines, and you can see that all across the board we're seeing tremendous improvement. So you know from 96 to 100 improvements on the connection.

A

Timeouts and the current teams are being reduced by 50 to 81, um and this is in terms of active curtains. So this is, you know, tremendous results. All right. uh Next up is just some tangible error data. You know we can see that we had 51 errors in a span of five minutes that got reduced to two errors um for the broken pipe.

A

We had a 51 connection, timeout errors in the span of five minutes when we're sending logs 40 megabytes per second one kilobyte logs that got reduced to one in five minutes, so tremendous results, um the current teams went down from 15 to seven on average.

A

You can see these errors. This is the sort of graph that we showed before tons of clusters of errors, um and you can see that the clusters are always preceded by these large amounts of proteins, and this is all 1.8.

A

So high number of proteins yields a high number of errors. Well now you can see the number of proteins went down, so this the axis is a little bit different, so the old axis was up to 60.. Now the new axis is up to 18, so these proteins are being kept at a much lower level. You can see that because of that there's much less errors. In fact, we only see three errors. You can barely see it down here at the bottom right hand, corner of the graph so yeah.

A

So again, just looking at these two graphs, you can see there's a tremendous improvement from 1.8 to 1.9.

A

um So just this quick summary, uh the network errors got reduced from 96 to 100, so really good, really good stuff. The active proteins got reduced by about a factor of three and then error clusters were removed. So all this is really great stuff. Okay, so um really quickly. Let's take a look at we're going to skip through the uh improvements and problems uh just so that we have time um all right, so we're going to get into the related domains of development, so um yeah.

A

So what can we do in the future for the event, so we covered so far the uh you know some some adventure programming paradigms we covered so far. Some uh you know problems with the event leave solutions with the event lube and also what is the event loop and and also how did that you know how did those changes, how this changes the event loop uh impact fluid bit in a profound way to make network stability uh improve and increase, uh but now, let's get into some related domains of development.

A

You know, what's left to be done well, if bluebit really wants to be as performant as possible. In order to do that, we need to utilize as many you know, threads as possible. uh So recently there was a transition from you know, just kind of single threaded application to multi-threaded application with workers.

A

So that means that every worker gets run sort of on a different uh server for output plug-ins, you can run output plug-ins on different workers, which, which get run in different threads.

A

That doesn't work though, however, yet for input, plugins or filters so kind of on the future roadmap flip. It there's a lot of talk in the community, about adding worker support for input and also filter, plugins, and so to do that we really have to take into consideration the event loop, um because you can kind of think of every single worker is having its own event loop.

A

So every time you have a thread that can run asynchronous code, it needs its own event, loop and so in order to make these changes, we'll have to have this fundamental sort of concept of the event loop in our minds which yeah, hopefully, this discussion sort of allowed you to to sort of uh gain the insight into all right. The next part is the reducing events so another you know key to this event. Leave system is to not have too much event pollution.

A

So if we have too many events in the event loops, then it becomes very inefficient. uh So there are some events right now that that don't really need to be there that we can reduce duplication of uh mainly like the dns timeouts and some of the other timeouts.

A

At the time when the dns timeouts were implemented, there wasn't really a good mechanism to coalesce these events, uh but ever since there was some code that was written, that brings together all the cleanup functions and that's used for like the network timeouts.

A

So some some talk that's been happening in the community as to bring together all these dns timeouts. So you don't have all these random events sitting everywhere. One for every single dns uh call uh one for every single network event um in order to really make a little bit more efficient because less of the less events that we have, the better that's the less person switching for the desks that we talked about all right.

A

So that's all that we have time for um so I'm really looking forward to the q a um right after this- and also you know uh just thank you so much for your time that you're spending to listen in on this discussion. I hope it really helps to give you some fundamental understanding of the flip event loop and how you can join in on some optimization efforts with this mental framework in mind.

A

Now, if you would like to stick around we'll, have a follow-up section for this to go into the details that we skipped over and also some deep dive on discussing kind of how exactly we implemented some of these changes right.

A

So the bucket cue um some task pipes, um some you know, multi-threading, uh you know between uh the different, the different threads and different workers getting into that just a little bit, uh but we don't have time for that in this discussion, so we'll post a link, maybe in the q, a for the full talk over there. If you have any questions, my github handle is matthew, folla, his first name and last name, there's spaces. No, all lowercase, please feel free to reach out to me uh and yeah.

A

I look forward to what fluidvit has to offer to the community and to all the customers going forward um and with all of the you know, network improvements that have been made and can be made in the future. um You know via the event loop um being fully optimized, and so thank you so much for uh joining us today and I look forward to our q a right after this. Okay.