Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes CronJobs - Does Anyone Actually Use This [in Production]? - Kevin Yang, Lyft

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes CronJobs - Does Anyone Actually Use This [in Production]? - Kevin Yang, Lyft

Considering CronJob? Think again! CronJobs sound great on paper–a higher-order API built on the bread and butter of K8s that automates fault tolerance, orchestration, etc., for distributed, repetitive tasks. On top of that, you benefit from efficient resource usage when your cron is idle. Who wouldn't want that? At Lyft, we run hundreds of CronJobs on our platform; however, the road to get there was not easy. In this talk, Kevin will dive deep into the guts of CronJobs, discuss subtle issues like how the control-plane can get bogged down at scale, and share how we addressed these problems in our K8s stack (including forking and patching Kubernetes itself!). You will leave with a better understanding of the trade-offs of CronJob and what features to look for in a cron solution to make your developers (and you) happy.

https://sched.co/ekDF

A

Hi, my name is kevin and welcome to my coupon talk uh today, we'll be talking about kubernetes crown jobs. Does anyone actually use this thing in production? Well, the short answer is that, yes, people do at lyft, we run um some 500, plus crown jobs in our kubernetes clusters. Currently, and um in this talk, we'll talk a little bit about what's been like, um but before we get too far into that. Let me start by introducing myself.

A

My name is kevin yang and I currently work on the compute platform team at lyft, where we work on building out our kubernetes kubernetes environment, that powers all of this services and compute needs.

A

This has been a multi-year journey and there's been a lot of things that we've learned along the way and that we're excited to share with the community. um If you're interested in getting contact with me, my twitter and email links are posted in the slides.

A

Alright, let's get into it. So what is this talk um in this talk I'll?

A

Tell you some stories about chronic failures that we've experienced operating um a crime platform using kubernetes at lyft along the way I'll, try to surface and poke holes at some of the flaws of prawn jobs that we've seen from both the technical perspective, but, more importantly, the user experience we'll discuss kind of how we moved how we smooth out some of these rough edges to deliver a better experience to our engineers that use our platform and, finally, we'll discuss some of the broader lessons that we learned um operating in such a platform using communities and what this might all mean for you.

A

So why should you care? Well, maybe you're like us, and you run a kubernetes platform at your company and you might have dozens or hundreds or even thousands, of developers that run compute workloads on your kubernetes clusters. Maybe you use cron jobs directly. Maybe you run, you know lots of crown jobs just like we do, um or maybe you're just looking for some info on distributed.

A

um Cron scheduling, repeated schedule, task platforms and you want to see what kubernetes has to offer.

A

Well, regardless of all these things, one of the main things that you should care about is what the user experience looks like for people who use kubernetes cron jobs. In particular, we found that lyft are you know one particular point, one particular workload type in kubernetes that hasn't got as much love as some of the others, um and I think there's a lot that we can learn um from kind of dissecting the issues and um and figuring out. Well, how could we can make it better?

A

But first, let's start with some story time.

A

It's generally good practice to have observability for your cron tasks, to make sure that you're alerted when they fail at lyft developers, do so most commonly by emitting metrics in their application code and use our alerting system to page the on-call when those issues happen.

A

So let's imagine you're a developer at lyft and you're just paged for a cron job failure, and it is your job to investigate. What's going on, so you know a little thing or two about cron jobs and kubernetes, and you use some tools to start inspecting what the status of all these different objects are.

A

So, first, you might do a keep. Color get cron job, maybe look at its event status and then from there you can see that the cron job has spun up a job object and from the job object you can see that spun up a pod object to actually run your application code.

A

Well, the problem is, you know, you're, probably at the point now where you're looking at these kubernetes objects, and you just got paged for failure and by the time you actually look at it. The pod is already gone. um Those of you who have some experience working with kubernetes know that this is far too common um and a pretty big annoyance when trying to debug issues with applications like kubernetes, because pods are vulnerable oftentimes, um it's hard to look at them because they've already been terminated, so you can't look at any pot events.

A

You don't know what the status of your containers are. So what can you do?

A

Well, I guess you could look at your application logs if they're you know log somewhere out to a log pipeline, um but more often than not you're kind of stuck, and you have to kind of wait for the next time the crown runs to catch in the act.

A

Meanwhile, you might have asked your ops team and the platform engineers are probably scratching their head as well. You know there's nothing that they can do either, and so you really have to wait until next time the crown runs catching the act or worse. The crown job failed too many times in a row.

A

Maybe this happened overnight and you just got paged for the morning and now we'll start it all, and you have to go into kubernetes and explicitly tell the job to run. This is a pretty um pretty annoying thing to have to do, and it's definitely something that we've run into lyft several times now.

A

Here's a different scenario that happened once um we had an incident that took down our crown environment for several days and took a lot of time and engineering effort to debug and figure out, but there's a lot of interesting things going on here that helped us learn more about, what's going on with crown depth under the hood and will help show case some of the drawbacks and technical issues that impact crown job performance.

A

So this started out this instant started out with a lot of user reports started coming in, um we saw a lot of users that are starting to report issues with their crons failing to run intermittently and now. Anyone who has worked at a company large enough to handle these support requests knows that um there's varying levels of detail that users provide, but we had some details. We had some users who actually put in a lot of effort into some investigation and that kind of helped us uh eventually recause the issue.

A

So one astute user actually filled out the ticket form and attached some of the investigative work that they've done.

A

Specifically, they showed us this chart, which um is a chart of log volume and every time the crown runs it logs some messages and for the times when it didn't run, there's obviously no logs emitted, and so we saw a chart. That looks something like this. Where you know there are several iterations of their crown job that emitted their logs and show that their application code was executing, but every so often there'd be a hole or gap um where there are no uh logs emitted, which indicated that the application code didn't run.

A

Now, as a platform engineer, we looked at that, and you know we thought is this something that happens on all crons or just a few crowns and sure enough. Most of the crons that ran on our platform uh worked fine. They did not have any such issues, but there were some crowns where we saw the same kind of log indicator that showed that application code was not running.

A

So what gives at this point, we started looking into what is actually happening with the quantum controller under the hood and one of the things that we did was um looked at the log level of the cube cube controller manager, which runs the cron job controller.

A

Some background, on the quantum controller, the way that it works is that it runs a sync, the world operation every 30 seconds, and so what this does is essentially there's one big loop in the code that lists all cron jobs, every 30 seconds and iterates them iterates through them, one by one doing whatever reconciliation, whether that is invoking the cron job, doing some sort of bookkeeping or whatnot.

A

So, when we're looking at this loop, um we started noticing something interesting by printing out the logs of the contract controller, which was that um in our crown environment we had about you know 200 to 300 crown jobs at the time for the first 20 or so um iterations of the loop. That is the first 20 crown jobs that were processed. um We saw that they were all processed pretty quickly, the time to process them was really low, and um you know the contra controller was behaving as expected.

A

Where we started seeing some issues was after the 20th iteration, um we saw that the times are nearly 10 times slower to process crons for the remaining 200 plus crons that were in the list. um So what might explain this? Well, it was rate limiting what the crown job controller does in sync operation. Is it actually, you know, sends some api requests to the api server using the cue client that the key controller manager uses and what we noticed was that um there's actually rate limiting that was bogging down the chronic controller.

A

Sure enough. The defaults for the uh for the acute client rate limit lined up with the lubrication times we were seeing, um so it turns out when you run, you know 200 or more cron jobs. You know those times add up, and you start seeing loops uh that take longer than 30 seconds, and hence you start seeing crown jobs get backed up as the chronic control is bogged down by rate limiting.

A

So what does this mean for our applications? Our crown jobs? um Well, a lot of our cron jobs. Had this field called starting deadline, second set, and what this field does is essentially tells the crown drop controller to stop trying to run it if it has been delayed by you know the starting deadline, seconds duration, and so what we would see in an application is that, right before the crown was supposed to run, say t1.

A

um That was the last time that the cron job processed, uh that the control controller processed that prime job now the cron drive controller loop took, um you know, say longer than 30 seconds and so by then the starting deadline seconds already expired, and so the next time, the contract controller. You know inspects that crown job and determines if it has to run well.

A

It sees that the starting deadline seconds has already expired, and so therefore it doesn't invoke it, and so this is why our applications, um our crown jobs, sometimes saw that they had missed their schedule because deep down the contract controller loop had taken too long to process and had missed a particular scheduled time.

A

What did these experiences tell us? Well, we learned that crown jobs can fail in many surprising ways that we did not know about before the first of which is the two minimum starts condition, which is where, if a crown job has failed too many times in a row, then the kubernetes machinery ends up giving up entirely on trying to run them and requires human intervention. To start the crown drive again.

A

Next we saw that um api client rate limiting can be a factor when the crown check controller does the sync the rolled operation. It requires a lot of client calls to the api server and these can get rate limited and impact the speed at which the project controller can process and sync cron jobs.

A

This combined with starting deadline seconds can lead to some disastrous effects where cron jobs get missed entirely.

A

So, aside from the technical issues, we also saw that cron jobs in general are quite difficult to monitor, understand and debug. There's not a lot of observability that ships with crown jobs by default, and you have to know a lot about the system and the underlying behaviors of kubernetes in order to debug these and understand.

A

What's going on, as we saw through instance, it took a lot of effort and learning from the kubernetes team to be able to understand, what's going on here and be able to recause the incidents and further cron job does leak a lot of abstractions from pods and jobs. um There are a lot of knobs to configure, especially regarding retries and concurrency behavior, and it's a little bit of balance of strike um in order to not surprise your developers and your users.

A

We also saw that um failures are quite difficult to recover from. In the example of the on-call that got paged. Well, it would be nice if they could, just you know, rerun the cron job. If it failed, you know. Maybe there is a service depended on in their application.

A

From a platform engineering perspective, we really don't have a great way of understanding the performance of the platform. You know we got a lot of user reports of issues, but it'd be nice if we could be alerted on these and not have to rely on users to tell us that something is broken.

A

So what did we do about this? One of the things that we did was enhance the observability of crown jobs on our kubernetes platform by introducing trace points at various stages in the cron's life cycle.

A

So this can be things like you know when the crunch controller decides to invoke the cron when the application code actually starts when the application code finishes, whether it was retried or you know, exit successfully and at each of these points we emit metrics. So we can know um some details of the performance like the start delay. How long did the crown job take to actually start running application code from when it was expected to run or the runtime of the application container itself and the exit codes?

A

These are all things that our developers want to know and want to be alerted on, but didn't want to have to write and maintain themselves.

A

And so, as a platform, we were able to have all these metrics built right in by default and have alerts created for any crown job that was on board and our platform, and this allowed us to get rid of a lot of bespoke alarming and metrics code from our applications and our services.

A

Next was disaster recovery, so we saw earlier that when a crown job failed oftentimes, an on-call didn't, have really anything they could do about it. um So one thing we created was a run cron button which essentially allows you to round the crown ad hoc.

A

So this is useful for recovering from failures, as well as testing out new crown jobs and debugging existing ones. Instead of having to wait around for the quantum run, you can just hit the button and immediately see the effects of your crown running and your application code doing its work.

A

Finally, we fixed the long-standing too many miss starts problem in our kubernetes fork, so they never had to deal with stock crons again once again, this was a problem that we had seen a lot of times at lyft, where, after a crown, has failed 100 times in a row, then kubernetes stops trying to run it entirely, and so, by fixing this nerf work, we never had to deal with that situation again, where we had to manually reinvent crons to get them started again.

A

So now, you're, probably thinking to yourself that seems like a lot of work to get kubernetes crown jobs to a usable state. What does this all mean for me? Should I still try to use them? Is this the only option? There's a lot of nuance to picking technologies like a distributed schedule task runner here are some things to consider before making the same decisions that we did at lyft.

A

First, ask yourself: what do your users really need? The primary feature you should have figured out for engineers using your platform is observability based on our experience. Talking to dubs.lift devs mostly want to know, did it run, did it run successfully and how long did it take at lyft? We built these metrics and alerts into our crime platform, so all crown jobs get them by default.

A

These monitoring tools are essential to allow engineers to self-service and be able to operate their workloads, no matter the platform they allow people to have high confidence in the reliability of their cron without having to know kubernetes as intimately as platform engineers do and really help scale the operations side of a large engineering org by making it easier for people to debug issues on their own.

A

Next, you need to be able to run acron ad hoc to debug or recover from incense. No one wants to sit around waiting to catch a crown in the act and especially in today's micro services. World failures can happen and we must be able to fail gracefully and make it easy to recover from failure.

A

Because of this, it is essential that your crown platform has an ad hoc invocation tool. That way say a downstream service is down and causes your ground up to fail. At least the on-call is alerted using the alerts that we just mentioned and can rerun the job without having to wait for hours or even days for the machinery to invoke the crown again.

A

This is absolutely essential for say once a week crown jobs that do things like generate generate reports for your company. An added benefit of a tool like this is that it allows engineers to develop new cron jobs and observe them running, live in your staging and product environments.

A

As much as we try to make configuration simple, inevitably there will be some trial and error involved when deploying new code. For the first time. There's nothing worse than having to wait to see your new code come into effect, so having a tool, be able to trigger your new code and be able to watch it. You know. Work for the first time is really helpful. Helpful for developing debugging neucrons.

A

So those are the user-facing features that we focus on delivering with our crown platform using kubernetes crown jobs that lift kubernetes crown job may not be the right solution for you, depending on what your environment is like if you're starting from scratch, how should you approach evaluating cross solutions for your company.

A

When evaluating cron platforms, the first thing you should think about is the user experience.

A

Devs really just want to write and ship code and be confident that if there's an issue they'll be notified and when they're notified, they can easily figure out what went wrong and recovered.

A

So the very first thing they should do when picking prop platform is talk to your users, figure out what the workflows are like and what they need out of a cron environment, because what users might want might differ a lot from might differ a lot from what you might want. As a platform engineer.

A

But as a platform engineer, what should that experience be like? So the major thing that platform engineers are concerned about is the performance of the platform. How easy is the monitor the reliability of jobs on your cron solution? You want to look at things like failure rate time. It takes to start running application code.

A

How well it scales as more and more ground jobs are added to your environment, so these are all things that you want to think about. You know: are they built into your crown platform, or do you have to add some stuff in order to have those metrics and be alerted on them features aside? There are many hidden costs you might not think about when using a new tool like kubernetes crown jobs like how much do you expect your devs to know about communities and how much effort is it to train them?

A

How much are you willing to invest in smoothing out some of the rough edges of kubernetes, for example lyft? We already had a lot of infrastructure and tooling in place for things like observability and reliability, so we only needed to plug in some modifications into them in order to read the benefits and, lastly, along the way, there will definitely be a lot of incidents.

A

How good is your learning culture and how resilient is the business to instance? That will inevitably happen.

A

Hopefully the picture is starting to get a bit more clear as engineers we can often get pulled into deep pulled in deep into the nitty-gritty of technical complexity, evaluating tools and technologies from a technical perspective is certainly valuable, but more often than not, the challenging part is breaking the gap between human and technology.

A

At the end of the day, there will be other human beings using the tools you build and offer so think, a lot about how humans interact with your systems.

A

So, to conclude, not all hope is lost. Recently a cap was merged for graduating, cron jobs to ga, and it's really exciting to see some progress being made in this area, and I especially like how this time there's a lot more concern around observability and performance scaling of crown jobs so kudos to the kubernetes team and I'm definitely looking forward to playing around with the new crown job api when it comes out.

A

But all things kubernetes aside, talk to your users, have real conversations with engineers at your company and ask them what they really need from your infrastructure.

A

And lastly, kubernetes is no silver bullet, but don't be afraid to get your hands dirty anyways and try to bring it closer to something: that's usable in the real world, and that's it thanks for taking the time to listen to my talk on kubernetes crown job.lift.

A

If you're interested in working on challenging problems related to running kubernetes at scale, then feel free to send me an email and meanwhile check out our careers page at lyft.com careers for current openings thanks.