Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Long, Winding and Bumpy Road to CronJob’s GA - Maciej Szulik, Red Hat & Alay Patel, Red Hat

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Long, Winding and Bumpy Road to CronJob’s GA - Maciej Szulik, Red Hat & Alay Patel, Red Hat

The CronJob API just reached GA, and the new controller is solving all the performance and reliability problems of the past. Come join us to learn about the 6 year journey that got us here! We will talk about the people who sparked the discussions and delivered the initial implementation. We will cover all the major problems that users were faced to handle over the years. Finally, we will discuss the solutions and our gratitude to the users and developers standing behind this part of Kubernetes. Maciej (one of the co-authors of CronJob) and Alay (developer of the new controller) will try to squeeze as much as possible in 30 minutes: - From scheduled jobs, through jobs, to cron jobs - Evolution of the API - Known issues with the old controller implementation - Performance boost and improvements in the new controller - Possible future improvements

A

Hi, I am oleg madel.

B

And my name is mate, and today I'm going to take you to a long, winding and bumpy road to crown jobs. Ga the idea of cron jobs wasn't entirely new in the distributed world. Identical mechanism was already available in google's internal cluster management system borg, which divided workloads in two basic categories: long-running services and bad jobs.

B

For the remaining part of this presentation, we'll focus on the latter, it wasn't long after kubernetes was officially announced by google in mid 2014 that this functionality will be needed in this project as well.

B

My first exposure to this topic was in the spring of 2015, when a bunch of us here at red hat, along with our customers and small community, that has gathered around kubernetes before its 1.0 release later that year. We started tinkering with this problem and writing down our thoughts.

B

First in openshift repository in may june of that year, eventually reaching consensus, I've opened a proposal back in august 2015.

B

only after we moved our proposal to kubernetes repository more people, including the authors of the borg paper, started looking at it. The proposal was sliced and diced into many directions.

B

Eventually, we've agreed to split the topic into two first, the original proposal morphed from describing distributed cron functionality to a primitive which allows running a task to completion today. This is simply known as a job resource.

B

This way, our focus slightly shifted and we dived into implementing jobs, which were part of 1.1 release of kubernetes.

B

It's important to mention the fact that back then kubernetes did not have api groups like it has today, so adding a new resource was challenging, to say the least, on top of that jobs and cron jobs were very often trade blazers in the areas of api groups and versions.

B

Finally, after shipping jobs in january of 2016, I was able to finally focus on the initial ask. What I haven't mentioned before is that back then cron jobs were actually called schedule at jobs naming his heart. Isn't it honestly?

B

I can't remember the reasoning behind the name, but I'm pretty sure it is still there in those proposals and discussions at that moment in time, we've all had a pretty good understanding of the topic, as well as the primitives and mechanism to build scheduled jobs. So as soon as the proposal was merged, we jumped into the implementation.

B

Even though we initially targeted 1.3 release, we had to delay the functionality until 1.4 due to many challenges. I've mentioned before this way. In the late summer of 2016, we had scheduled jobs in kubernetes.

B

There is one important cavite which I have, which has a crucial impact to the rest of this presentation. As you see, the scheduled job controller was written in 2016. back then. Writing controllers was completely different from what it is today.

B

The controller was periodically pulling all scheduler jobs and triggering the ones that were supposed to run this.

A

B

Seemed the simplest and most obvious to solve the problem at the time after that initial sprint, both with jobs and later with scheduled jobs, we slowed down and focused on other features and fixing bugs the two major developments worth mentioning here were when we decided to rename the cron jobs which eventually landed in 2015.

B

The biggest challenge was that for a few releases, we actually supported both the old scheduler jobs and a new cronjobs name, which wasn't an easy thing to do.

B

Eventually, in 1.8, we've decided to promote cron jobs to beta without too much changes in the api or the controller that was back in 2017.

B

Until 2020 hit us from many angles, for us, the biggest impact was the proposal to remove all the so-called perma beta apis from kubernetes.

B

In the near future, the community gathered around special interest group devoted to running application or sigaps, in short, debated the problem for every api, where we've been impacted, since that only cron jobs had a similar issue of perma-bait state we've had many lengthy discussions, which you can check in the sig apps meeting, notes and recordings, the outcome of which was that the best way to move forward is to reduce to a first introduce a completely new controller, gradually switching the controllers through a few releases and only then promoting cron jobs api to general availability.

B

This would ensure that the existing production clusters wouldn't be immediately affected by these changes. The main reason behind all this was coming from how modern controllers are written. The previous controller was periodically polling api server. Modern controllers, on the other hand, are only notified about changes to resources.

B

The seemingly slight change has a significant impact both on the performance, but also on the scheduling algorithm of the controller.

B

But more about this, you will hear from ally who approached me last year expressing his interest to write that controller.

A

As much you mentioned most modern day, controllers have evolved into a notification based system to synchronize state. Let's take a brief look at what the architecture of a stock controller in code kubernetes looks like this will help a lot in understanding how the new cron job controller is implemented.

A

As you can see in the picture, there are two key elements to a cube controller that are connected by the queue on the left. We have the shared informers giving us the add, update, delete notifications on the right. We have the other element, one which is responsible for performing the sync actions upon any event from the informer, the resource event. Handlers allows us to push um keys or objects to the queue the handler on the right, then pulls those objects from the queue and performs reconciliation.

A

The queue here makes sure that no two um handler workers are reconciling at the same time, on the same key, this allows us for safely scaling up the number of worker handler functions.

A

The informer also acts as a cache and allows the controller to list or get objects from it. It is quite clear that if the new cron job controller is implemented on this architecture, we would improve the performance by reducing the number of calls to api server. Also, we will think we would improve on the scalability aspects now, let's quickly dive into and look uh at some examples of the informers and event handlers.

A

So if you look at the code snippet, this is the cron job controller, using the jobs informer to register three event handlers. Typically, a controller works on the set of objects. For example, the cron job controller cares about events from the jobs as well as crown jobs in order to cue the cron jobs effectively, it uses both the crown jobs, informer and the jobs informer.

A

The court snippet shows this um shows the ad job event handler further down the slide. You can see that the edge of event handler takes in the um object that is coming in via the event um type casts it into the job object. It then uses the job's owner reference to find the appropriate cron job for it. It then enqueues the cron job onto the queue.

A

So to sum, this up resource event handlers along resource event. Handlers are implemented in a way that converts the events from the informers into specific keys that are inserted onto the queue once we have the elements on the queue the wiring of the workers or handler um handlers that perform sync operations is actually quite simple: the worker will then just um pull um or loop on a function called the process. Next, I work item function. um The very first thing in that function is the function is attempting to get the object from the queue.

A

This get call is a blocking call. It will block until there is an element in the queue once the worker has the right key to process from the cube. It will just pass that key to the sync function. The sync function is where the business logic of the controller is once the sync function is done. This call process to process next work item returns and then um the loop again blocks on the same call in the queues cat function.

A

Looking at the slides more closely, one may wonder what exactly is the req after variable, that is being returned by the same function. This is actually something unique about the new cron job controller and is a nice segue into um looking at the scheduling aspect of this new control, along with the updates of the api server. In order to implement this new controller, we also need to handle the scheduling aspect, for example, my back um in this architecture that I just described couples of slides ago.

A

The only way a worker performing the sync operation can be invoked is if there is an element in the queue. What if there is no add update or delete event around around the next scheduled time of a cronjob? What will push the cron job element onto the queue around the scheduled time? uh There is no guarantee in this architecture, so we have to handle this case of scheduling.

A

Then, as you can see in the picture, this is actually handled by by the workers themselves. In the new controller's implementation um anytime, the worker is done processing a cron job. It will return the req after the time interval, after which the next schedule for this cron job is supposed to be triggered. um It uses something called a delaying interface.

A

The cue we use for this new controller implements this delaying interface. This allows us for inserting elements into a queue into the queue after specified time interval. So the time interval returned by the sync job sync cron job function will then be used to requeue the cron job object appropriately at the right time on the queue.

A

This is how we handle scheduling for the cron job now, once the the enqueuing of elements onto the queue is taken. Care of the next part that comes into um into this new controller is how it implements the sync function. As I said, this is where the business logic of the controller is the very first thing it checks is if the cron job is suspended, if it is indeed suspended. This is a new op for the sync function and it simply returns if the cron job is not suspended.

A

It then looks at the most recent scheduled um time after the last scheduled time.

A

It then says if the current time is already missing um the deadline for this most recent schedule time, if, if it is missing, if it has already missed the deadline, the controller cannot do anything.

A

It simply calculates the time difference between next schedule, time and the current time and returns this sdq after so that the cron job can be enqueued properly on the next scheduled time. If it has not missed the deadline, it will then go on to check the policy for this cron job.

A

The current policies can be allowed, replace or forward depending on the policy. It will take different actions. These are simple switch case or, if else conditions in the controller, you can go check at the sync current job function. To know the exact specifics of how it is implemented.

A

Note one note is that, upon suspension, the same function does not bother to calculate the time difference and enqueue appropriately based on time. The assumption here is when, when the crown job is suspended, unsuspending it means that we would have to update the crown job and the event will come in via api server through the informers and the resource handler resource event. Handlers we don't need to handle that case inside of the workers itself.

A

Also note that the sync functions- the sync function that I just described always performs the same um for the schedules that are older than the current time. uh It re the it always tries to reuse req for the schedules that are newer than the current tank. This distinction is important to understand the workflow.

A

Now that the basic workflow of this new controller is established, let's look at some of the corner cases that come in here.

A

One of the challenges is um how to handle updates um it's one of the harder things to work through and wrap your brain around first, but once it is done, um it is actually quite simple to understand.

A

The interesting part in the updates, if what, if the schedule is updated, we already have the key for this updated cron job in the queue that is reflecting the order schedule. What happens to that key? um These are the kinds of questions that pop up um so anytime. There is an update or change in the schedule. We use the update, cron job resource event, handlers handler um to enqueue for the next scheduled time of the newer schedule.

A

uh This could mean two things, uh one that the next scheduled time of the new schedule is um is earlier than the then. The last process scheduled time or the other possibility is that the next scheduled time is pushed further back and is later than the last process scheduled time.

A

These are reflected as two possibilities in the picture: the t1 minus delta t and the t1 plus delta t, of course, t1 being the last process scheduled time of the order schedule.

A

So these two possibilities um can occur in the for the the second possibility, where the new schedule is pushed further back in time is actually easier to handle. The workout will still be fired at t1.

A

It will determine that it has either missed the schedule or already has a job for the last schedule and do a new op. It will then calculate the delta t and uh push the key back um into the schedule accordingly.

A

In this way, we we make sure that we have t1 plus delta t um at t1, plus delta t. We have a key on the on the queue to reflect this update.

A

If the next schedule is earlier than what the controller has processed the update, cron job resource event handler will see this change and push the key for an earlier time. This just means that the earlier uh earlier time key will will be fired first and then we have, we will have a worker that is fired at t1 minus delta t. So these are the two ways in which um we handle the update of schedule. um It was one of the interesting things to implement when reworking on this architecture.

A

Let's dig a little bit at on another set of challenges that were introduced due to cash.

A

One of the ways that we were we are able to increase. The performance of the system is to um leverage the cash. The classic problem of stale caches can manifest here as well. The cron job informer and the job informer are the two caches that the new cron job controller uses, and it can be easy to imagine that if one lags behind the other, we could have, uh it could cause some problems.

A

For example, the crown job controller creates the job for this schedule and puts it in the crown job status in the active list. This creates multiple updates, um one update at the job, informer and one at the ground dropping from. As you can see in the um in the picture, this updates directly go to the api server and then the informers would be able to see this. Depending on the exact scenario. It could be possible that one informer sees it before and the other informer is still a little slow and does not see this update.

A

So the specific example illustrated here is that, let's say cron job informer sees that update. First, its status will reflect an active job, but the job informer is still slow. It did not see the update yet so when the controller tries to go, get use the lister and get the job object from the cache it will. It will get a 404 not found error.

A

This could, if this case is not handled appropriately. This could mean that the controller would take improper decisions because of the lacking cash problem. For example, it could create a duplicate job for the same schedule.

A

The way we handle this um in the in the current implementation of the new controller is anytime. There is a potential problem identified due to lagging caches and it could cause um it could cause behavioral problems.

A

We actually go to the api server and perform a live, get call against the api server to get a fresh copy in the example earlier, if the, if the job is not found in in the informer, we would make a live call against the api server to see if the job is actually missing or we are hitting this stale cache problem that way. If, if it is found in the api server, we would process it accordingly and not cause it to create a duplicate.

A

There are two instances where this could happen and the current controller handles both.

A

The lagging caches are not the only problem that cash introduce. The other problem is is actually quite easy to encounter. um As I mentioned earlier, the controller uses two informers which are caching objects. Now these informers can be shared across all the controllers um in the controller manager. Hence they are. Hence they are aptly named shared informers right. So a caveat to this is the objects pulled from this. Cache should never be mutated in place. It is, it is actually one of the. As I said, it is actually one of the easiest mistakes to make.

A

While writing on a controller.

A

This has made it um into the list of community guidelines. For writing the controller. There is a link to that page in the slides. You can go, um read it in that.

A

The solution to this problem is actually very easy. Anytime. We want to mutate. The object from the cache just create a copy of the object when mutating that way. That way, other controllers will not see what you have mutated, unless you explicitly submit it to the api.

A

Server, so what do we get after? um All of this uh one should ask, after the entire um re-architect and over, um overcoming all the challenges. um What are the advantages um we should expect? So, theoretically speaking, we should get less calls to the api server.

A

This should make the cron job controller more performant, because less time is being spent on waiting for those network calls. This should also put less stress on the api server, so the overall system is more performant.

A

It is also really easy to scale up the number of workers so assume the case where user knows that they have a lot of cron jobs in their cluster and the default of one worker is not sufficient for them. The user can safely increase the number of workers and they could achieve this scaling up of the controller.

A

So how do we go about um testing this performance um improvements? um In order to see the improvements? We actually ran. um Some stress tests on it. um We used a vm with 128 gigabyte of ram and 64 virtual cpus, a really beefy machine um to create a single node cube cluster. We wanted to be intelligent about what kinds of crown job we create so that we don't overload the um entire system, so 20 cron jobs were created that were to be scheduled on every minute.

A

Additionally, to stress test the controller under duress, we would create another 2 100 cron jobs, with the schedule of running at every 20 hours. This would mean that the controller will still have to process all the cron jobs, but it does not have to create the jobs or the parts for this.

A

This will limit the number of pods concurrently running in the system and potentially stop overloading other parts of the system like the cubelet or eps server.

A

We would additionally create batches of thousand cron jobs until the the total reaches to 5120 crown jobs.

A

With this sample workload, we will change the controllers with the feature flag and compare both the old controllers performance and the new ones.

A

The old con, the old controller, would really start to show some performance problems when adding cron jobs in thousands for every thousand additional cron job added in the old controller we would see- and it requires an additional two minute of delay in scheduling that those 20 jobs it's supposed to schedule every minute, that is, for 2120 cron jobs. Instead of creating the the 20 jobs every minute, it would take the controller about three minutes to create those jobs.

A

This is this is quite drastic when the workload was increased to 5120 crore jobs about eight schedules were missed um before the next 20 jobs are created, so it would take about nine minutes for the old crown job controller to um resync of the 20 crown jobs.

A

This linear increase be in time between processing consecutive job creation, um when adding more and more cron jobs actually shows the scaling problems of the old controller.

A

Comparing this with the new controller, it did not really see any lag in job creation. We actually went ahead and topped it up with additional thousand crown jobs, making the total to 600 6120 cron jobs, um and it still did not see any visible um delay in scheduling those 20 grunt jumps now. Obviously, this is not a um a great real life model, but it actually um the sample workload actually shows um how the new architecture is more performant under duress and solves some of the scaling problems of the old one.

A

Along with the architectural change, there are few smaller new additions going into the controller. We now have a new histogram metrics um that users can go up. Look at this metric.

A

This matrix shows the skew between the job between when the job is supposed to be scheduled and when it is actually created, we have more metrics planned for this controller in the future. There are also minor optimizations.

A

One notable example is the compute next schedule. It was observed that um the old controller will would um use a library function to collect all the scheduled time between the last schedule and the most recent schedule and store it in list, um given that we only need the most decent schedule um and the granularity that the crown job supports is one minute. It is actually very easy to replace this for loop and use uh math to calculate the most recent on schedule.

A

This optimization actually helped us in saving some memory requirements.

A

With this new controller, we were able to drive the cron job api to general availability with kubernetes 1.21.

A

Lastly, I would like to thank the seagaps community for allowing me the opportunity to work on this controller. It was, uh it was an amazing ride. Hopefully, this also makes the user experience of cron jobs better. um If you guys have any questions we can take them now also feel free to reach out to us offline. Thank you.