Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Thriving With Kubernetes On... - Sunil Shah & Ramya Krishnan, Ashley Cutalo, Madhu C.S., Fabio Kung

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Thriving With Kubernetes On-Call: Best Practices & Lessons Learned - Sunil Shah & Ramya Krishnan, Airbnb; Ashley Cutalo, Lyft; Madhu C.S., Robinhood; Fabio Kung, Netflix

Speakers: Sunil Shah, Ramya Krishnan, Ashley Cutalo, Madhu C.S., Fabio Kung
Kubernetes clusters are critical infrastructure at large, public companies, with large amounts of traffic, complex dependencies on 3rd party services, and constant change as developers release features and traffic scales up and down. In this panel discussion, engineers from Airbnb, Lyft, Netflix and Robinhood share their challenges, experiences and learnings when it comes to managing a sustainable on-call rotation that meets the needs of their internal users whilst maintaining a high uptime to serve business critical workloads. Topics covered will include: +Keeping on-call engineers happy + Balancing rapid response with alert fatigue + Strategies to proactively deal with production issues + Preparing engineers for on-call

A

Good afternoon Welcome to our panel discussion, my name is Sunil Shah and I'm, an engineering manager within the compute infrared team. At Airbnb we manage all the kubernetes infrastructure Airbnb and at Airbnb we've been running all of our production web services and kubernetes since 2019.. We're excited to discuss on-call best practices for kubernetes with you today. Each of the panelists here operate kubernetes at high scale in production at large technology companies. Let's start with a brief introduction by each panelist.

B

Hi I'm Ashley catello I am the tech lead of lyft's application. Runtime organization, which comprise kind of your foundational infrastructure components such as compute networking data stores, I've been at list for five years and Lyft has been running kubernetes in production at scale for around three years. Now our workloads are ranging everywhere from stateless Services, hateful Services ml training on gpus Etc.

C

Hi everyone, my name, is Fabio Kung um I've been on Netflix for almost six years now. I am a staff engineer at the compute team in Netflix. Titus is our container platform um and it's been running. It predates me, so it's been around seven years around and it was built on top of mesos before um we ran. We migrated the gut soft titles kubernetes around three years ago, um mostly with not mostly without people even noticing. So there's a big thing for us. Move from mesos to kubernetes I've been running kubernetes.

C

Since then, titles is running a variety of all the workloads on Netflix, ranging from sales web services, stateful Services, real stream, real uh real-time data, infrastructure pipelines and machine learning, workloads encoding as well, hello.

D

I'm madhu I'm, a software engineer and Tech lead for the orchestration team at Robinhood. My team is responsible for running the compute platform for all of Robin Hood. We have been on kubernetes for a little over three years now and but I have been at Robinhood for only a little. Over a year now, before Robin heard, I worked at Google for about eight years of which I spent a little over half of the time working on kubernetes itself and and personal related Technologies.

E

Hi, my name is Ramya I'm, an SRE embedded with a computer team I've been here for around five years, uh I've been managing kubernetes for around four years since its in section uh within Airbnb.

E

um My team is responsible for kubernetes upgrades hcd, upgrade managing Auto scaling groups, uh managing cluster Auto scale or certificate management and cryo and the whole nine yards um yeah. It's a pleasure meeting you all so.

A

Let's, uh let's start with a brief overview of how Uncle operates at each organization, um specifically, what is the split of responsibilities between teams using kubernetes and your team operating kubernetes, and how do you organize your on-call rotation within your team? Let's start with Fabio.

C

I use is completely abstract in kubernetes away from our users. um We don't expose the kubernetes API internally at Netflix today and we package kubernetes as a product, so we offer a job in task-based API, so people can run workloads based on that API and we fully own that service. We operate it as a product as internal product, so we have a clear interface clear contract between what's who's using uh what we offer and they don't know about kubernetes, it's an implementation detail from for them.

C

um We have full control over how we use kubernetes, so any any problems, any issues, it's completely in our scope and our own calls right now organized in week, long ships we have a primary and a secondary rotation secondary false primary. So each person is roughly two weeks on call, firstly secondary then primary um and we have a single pool for everything, uh all the all aspects of the container platform, all aspects of title. So it's all people are on call for everything.

C

Today, the single two big large rotations primary secondary, and we tried splitting in the past. um We now combined and moving back and forth It's the constant trade-off for us on creating silos or managing siloine inside our team and managing also the surface area. As we combine there's a lot of logic over um uh I, wouldn't necessarily say we got it right yet we'll be iterating back and forth, um and we're likely to continue iterating we're trying breaking up a little bit again. Try and combine it a little bit more.

C

We'll keep it trading on this. So.

A

Thank you, Fabio.

C

D

Want to go next, our approach is slightly different to this. We have a platform experience team, that's different from my team that provides higher level abstractions on top of native kubernetes apis, and we call those apis archetypes, but those apis are also essentially just crd. So it's very much a native kubernetes experience platform experience team is also responsible for providing a UI.

D

All services at Robinhood are expected to use those apis and UI orchestration provides the foundational compute platform for the platform, experience team and the layers above um so when it comes to orchestration itself, orchestration is the team, that's response, which is my team. It's the team, that's responsible for managing the life cycle of kubernetes clusters and any other underlying AWS infrastructure. We are a full AWS shop right now. All of this is in addition to developing and shipping in-house infrastructure software on top of kubernetes.

D

So in a sense, my team orchestration builds and operates an internal distribution of kubernetes that supports the company's growing needs. So during instance, we call themselves for Service events like many other companies in the industry. Service teams make a quick determination as to whether the problem is at the application Level or at the infrastructure level. uh If they're determine that, if it's an infrastructure problem, they page the appropriate infrastructure teams on call, they can go into this tool called as the girl Services tool where they can look up.

D

The mapping between infrastructure services and on-call aliases and then page that on-call Alias, for example, platform experience on calls page. If the issue is an archetype orchestration on call is based, if you think it's a kubernetes level problem, but unfortunately it's really hard for application or service teams to determine that. So orchestration is sometimes considered a catch-all and we get a lot of pages. Orchestration itself is divided into three teams: container orchestration, compute infrastructure management and Cloud infrastructure, and the appropriate team is engaged based on the problem type and.

B

D

There is a human router involved if the teams cannot determine which exact team, the uh pages should go to orchestration incident. Responses are 24 cross seven three day, rotation is a single rotation. We have experimented with various duration, but this has worked well for us. There is also a separate business as only three days: service, desk rotation, wonderful.

A

Thanks um yeah I'm, seeing a bit of a theme here, which is um you know, for each of your companies, you've abstracted away how users can actually access kubernetes, which I think is um definitely something we've considered to Airbnb. um Actually, I'd love to hear about your experience. Yeah.

B

So at least we also go the abstraction route. I think we are what I call aspirationally serverless, where I'm not quite there yet, but uh we do aim on the infrastructure side to abstract away implementation. Details of our infrastructure from service owners, and so service owners are running on top of our kubernetes platform, for the most part are responsible for justifying their own business logic based slas and getting paid based on that.

B

The compute team is then responsible for the overall health of the kubernetes infrastructure, so that's kubernetes itself, um ec2 and anything that is running on that layer. And then we have another team called control systems that operates the operators that generate objects for services, service owners. That kind of manages the platform layer that the service owners interact with there is a bit of a gray area and I think this is the area where it becomes difficult in that kind of breakdown doesn't really work anymore, which is operator owners.

B

We do have a few teams that deploy and manage custom operators that are not within infrastructure at Lyft and the way that we've tried to Wrangle. This is that we try to spell out in advance in explicit contract and escalation policy, for these teams.

B

That kind of show that you know their level the first level of support where you know issues from their customers come in, they triage them via the platform and then, if it, it can be proven to be an issue with infrastructure layer below that, then they should escalate and involve the compute encore.

A

Thanks Ashley, um it sounds like there's, definitely a theme Here of kind of um escalation between teams, depending on where we think the issue might actually be, and that's definitely something that seems a little bit messy with uh you know with kubernetes in general, just because the surface area is so large.

A

um Ramya um tell us about your experience at Airbnb, yeah.

E

uh We also have a listener. Looking we have something called OneTouch API, which is an internal tool that we have built that helps the product things both services at one kubernetes. They just start away all the yaml files that need to get applied to build Enterprise Services into kubernetes. uh There's a lot of that. We maintain at your API and there's still a lot of questions from our product teams about these apis. So to help them we have office hours a weekly offers our rotation. We have a stacked holder flow like implemented.

E

It's an act, overflow implementation, where people ask questions and then get answered, and we have a good developer portal or documentation that they are expected to refer before, asking all the questions everywhere else. We also have clear expectations: all modes are ephemeral. Everything will go away 14 days, irrespective of uh whether you're stateful or stateless and teams are expected to handle these uh handle these rotations gracefully and not have incidents every time we rotate instances.

E

We also have high touch customers who have crds running and we isolate them into separate cluster, because once the RDA should not affect any of our other customers, so high-touch customers get their own special clusters uh within the equivalent. Kubernetes is a huge like huge infrastructure like huge surface area, so we are divided, broadly divided them into three parts. The other team members shift across We Have, a scheduling part. We have a foundation part and we have a one touch part that builds the apis, that our customers use.

E

The on call, as, as everyone said, is shared between a prime is shared across all these spots, and there is a primary on call and a secondary on call where the secondary acts like a fall back to the primary.

A

Thanks Ramya cool, so yeah we've touched a bit upon. um You know how the on-call rotations roughly look and how the split of responsibilities is um once you've got on, call up and running how? How do you all um you know, facilitate the sharing of state and knowledge between members of the on-call rotation, for example, there may be a production issue that carries over from one week to the next. How do members of the rotation communicate context with each other? Let's start with.

D

Madhu there is an orchestration on called slack Channel, where quick, handoff notes are passed from the outgoing on call to the incoming on-call person. That's mostly how we transfer State this typically cover. This notes typically covers the context about ongoing incidents and things to watch out for in general, in.

B

Addition to that there.

D

Is a separate weekly on-call review on-calls are expected to fill out a very, very short form per cell, that they responded to before this review and then on calls managers, TLS and other industry team members discuss these services in responses to these cells in the form. In the review meeting, we look for patterns and come up with short, medium or long-term action items during these reviews. This is aside from self-corrective actions or SCA is taken during the sales themselves, which are very specific to the scas.

D

We also have a company-wide tool called as Houston, which is part of a larger incident response Suite, where all incident responders are expected to log their hours and provide brief descriptions of the scenarios. This data is used for higher level analysis and for finding larger patterns across the organizations or the company very.

A

Interesting um and I can see a lot of parallels to what we do here. um You know, slack seems to be a big part of sharing context. Also, you know some other sort of instant messaging system and that's definitely worked pretty well for us. Actually, um how does it work at Lyft, yeah.

B

We have a lot of similar things like everyone else has been discussing. We do have the primary and a secondary setup at all times where primary is usually the one handling issues and secondary is not normally expected to have to respond to incidents, but is more of a fall through so to capture issues that will maybe take more than one week to resolve. There's kind of like two paths here. So there is a weekly handoff meeting that is part of the team's weekly planning cycle and there is a on-call summary doc.

B

That's kind of like a rolling summary that the on-call will fill out and pass on to the next for things that may be more like debugging, related or a feature request or whatever that we will get in from customer teams using the platform. These are things that we found that our on-call rotation was not a good fit for, because these things usually took more than one week and the customer did not like being handed off from person to person or things could get accidentally dropped very easily.

B

And so we actually created a separate role that sits in a in a team called infraops that is responsible for Less urgent, but probably more long-running. Issues like debugging requests, feature requests, Etc and that enables these items to stay with the same owner until they're resolved cool.

A

Thank you, um Romeo.

E

Yep, you have a very simple setup as well. Well, we have a primary and a secondary on-call primary handles all the tickets and if there is any fall through the secondary, handles it as well, uh this is a weekly schedule and every other at the end of every person's shift. There is a one called handoff meeting and there is typically a Google doc where people maintain their major changes or major major uh major incidents that happened in the last week or small incidents that are continuing to happen and needs to be needs a further investigation.

E

uh We also struggle, but what incident needs to be like handed off and what incidents need to be like that? Can the person not call take it to completion, even though he's not on caller he or she's, not on call for the system anymore? We also meet in Ajira on-call tasks. uh We are trying kanban right now. um uh This is a list of low priority tickets and debugging tasks.

E

Requests that come during the week on call probably doesn't have bandwidth for this, and there is a separate rotation bug rotation for to look at these tickets and three out of these tickets and like take it education, yeah.

A

And I will add that the bug rotation is actually a great way for people to onboard onto their own core rotation, because um you know it allows people to tackle on-call like tasks without the time pressure of being in an incident. uh Fabio.

C

I think this will be a bit boring, it's very, very similar to what everything has been said. um We also run weekly hand-off meetings and we use a rolling document, uh primary fields that documents I think one thing that's maybe unique, not super unique I heard that, before from other panelists, is that we merge support, Hong, Kong, instant response um monitoring, all that the it's it's part of our on-call rotation. So all the responsibilities are combined and we make sure we go through all that in our end of meetings. um We it's very.

B

C

I would say Netflix is that, but part of our freedom responsibility culture. So we lean a lot of the individuals in a lot of cases, for example between primary secondary. We don't have a lot of rules or structure on how do we will handle the load it's up to them, to divide and will balance the load as needed. Primary is always expected to be um responsible for initiating all of it, but the primary also needs to pull others as necessary.

C

um Aspirationally. We want everyone in the own culture be to have be comfortable with the huge surface area for everything we know in reality, there's always a little bit of a specialization, so we try and balance that too primarily responsible for balancing that um pulling others as necessary, especially with their specialization necessary.

C

um Another thing: that's maybe a little bit unique in what we do. We haven't heard before we spend time in our Handover talking about the health of the on-call, um so we we use a template and we always make sure we talk about how much time people spending handling pages of hours, how much risk that was perceived on the system.

C

What's their qualitative assessment of how was the shift, and it's just prompts for us to make sure that we talk about those things, they're not brushed off and we make so heavy Health on healthy on call. This is a high priority for us all. The time yeah.

A

That's a that's a great call out thanks, Fabio um yeah, the on-call is, is definitely a very stressful, um an anxiety-inducing activity um and, and that's actually a nice segue into the next question, um which is uh one of the conversations I have frequently when people join our team, um especially if they come from a team that you know. Is it a smaller company with a little bit less responsibility? Or you know, smaller clusters um is just a lot of kind of uncertainty and anxiety about joining on call.

A

um Kubernetes is a big surface area, as we've mentioned a couple of times today. uh People who are new to the technology or the team are often concerned about the their lack of experience and comprehensive knowledge and not being able to help when there is an incident. um So I'd love to hear how you all prepare new people to join The, Encore rotation and get them ramped up. Let's start with Fabio I.

C

Think we're still learning um while we're trying- and it's me working relatively well, but we're still involving this. uh It's not it's not great for people to join. There's a heat surface area like you're saying so we're yeah managing the situation. I would say um we do onboarding sessions and workshops. We recorded a bunch of those uh that's ranging from day to day. How do you? What tooling, do you use to get to troubleshoot things and to uh gather operational data?

C

So you can trace what's going on, um ranging from that to higher level architecture onboarding sessions on how things plug together, how they work? What what are the interactions on the systems? Where are the hot spots?

C

um We also typically new members, will Shadow other on calls um at least once but multiple times, even multiple. They will just uh add themselves to the rotation and they will pair up on on any issues and questions and support that happens, um so that helps build confidence as well uh in the opposite during their first or second shift 13 cup in the beginning, when they're starting to get into the involved rotation, we'll ask oh there's more experienced members to Shadow them as well to have a buddy um and yeah.

C

uh We have rambots, um though, as you may have you disagree, I don't know, but keeping run books is always a challenge. Keeping Room books up to date. It's always a challenge. um Would you have run box? We talk about them during handoffs um and being very supportive as a team I think it's important for us. We know it's a huge surface area.

C

um We can take another option. You can take three to six months for a member typically to feel confident joining your own call, rotation, yeah.

A

That makes sense um yeah. We usually wait at least three months for someone to join the team before they start preparing to join The. Encore rotation and I often tell people when they join a team like Alex. If it's like you know, you should expect to feel like you used to you're, still figuring things out for probably the first year, um there's just a lot to handle.

A

um Let's hear about lift yeah.

B

So this is also very difficult, operating kubernetes, like involves surface area and then also like massive risk to your organization. Since, if something goes wrong at this layer, it's going to be pretty severe, um it's not handled appropriately. So this is something where we find it's it's pretty challenging to do this correctly, in terms of you know, bringing people on board while minimizing risk to the company, particularly with very junior team members.

B

However, we did recently uh successfully add a new grad or entry level uh person to our rotation and it's been going great, she's doing great, and so it can be done.

B

um We find that we have the most success by starting with shadowing and so we'll have new team members Shadow like primary on call, meaning that they're they receive pages but aren't expected to be responsible for triaging.

B

But just following along and asking questions um after they're done with one or two shadowing sessions, then they will do reverse shadowing in which they and an experience on caller call are paged and the new person is now responsible for trying the triage but there's backup where there's someone there who can help them if they get stuck um and then only after they have completed one or two of those and it's going well, we will introduce them into the rotation.

B

Something we've found is that no one ever is going to feel ready enough um when they are put into the rotations a bit of a trial by fire, and so it's more about thinking about ensuring that they know where to look and how to get help in the event that something happens. They don't know how to handle and so that they can kind of learn on the job while not compromising our production safety. Absolutely.

A

And I think that's just the nature of Uncle like it's never predictable, because if it was predictable, ideally you would have automated that edge case away or that you know that scenario away so um yeah definitely challenging um around me. I want to go next. Yeah.

E

We have we have the same issues as everyone else had well. uh One thing we do is we have a lot of code. We have a lot of different code bases and we typically let the First on let the new body new hire, create a cluster which would involve creating uh PRS across multiple repos and learn the parts that change or get created during cluster creation.

E

uh We also, we also have video recordings of uh past presentations of different components of kubernetes, so these videos are recorded and is in some Google Drive and the typically new high algorithms and watches all these videos and then have a session have a session where they ask questions about various sub components into kubernetes, uh then they get to become secondary on call First and handle all the low priority tickets and uh and all the data tasks that come in as part of as part of that rotation.

E

uh Finally, they become primary on call when they are comfortable and typically every new hire has a new higher body. So then you, everybody and the new hire typically joins and collaborates basically for the first shift and.

B

E

Lot of help from everybody, as well as other team members and that helps them tight over the first on-call um yeah. We we have, we have run books, we hope to keep them up to date. It's a cat and mouse game every every now and then something goes off in the script and has to be a has to be updated retroactively. We also expect others to have clear instructions again that that also changes. So it's a uh it's again, a Canon Mouse game and then dashboards.

E

The same case like after every incident, I find another metric. That's super useful and added to the dashboard. So dashboards are super useful to triage on call issues, so um we just have a list of dashboards in a documentation that helps the uncle. um We also have Fighters and every now and then every Wednesday. We have a fight route that hopefully helps the new hire as well. During large-scale events, uh the on-call is not expected to be alone, most of them.

E

My few experienced team members just log into jump into the slack channel that is specific to that particular coincident and yeah. It's all hands are that there's an incident there yeah on corner I, don't think on call ever feels alone, handling large scale.

A

Thanks Ramya uh and finally, um madhu.

D

Yeah, our experience is sound very similar to particles, and in fact this has been one of our biggest people challenges right now, while huge surface area and familiarity with the technology are certainly challenging. People also come in with very levels of incident response experience previously after three to four months of starter tasks that included seas and startup projects mentored by an onboarding buddy.

D

We would ask new members to do a shadow shift and a reverse Shadow shift and ask them to go on call that was prone to be insufficient as we grow and hire people with different levels of experience and familiarity. We have our own books, but many incidents are like mystery. Thrillers Thriller is the operative words, because it requires some heroism from someone who knows something about a specific technology many times so due to that.

D

This is all due to the complexity of the Technologies involved, which requires to Deep dive even to mitigate the incidence right, not just root cause, so we are working on revamping our onboarding process right now. We are planning to introduce a curriculum for new team members to facilitate structured, self-based learning, something like what Ramia was saying about recorded videos. We haven't done that, yet we are working on them right now. More recently, we started running team-wide game day exercises deep dies on different areas from subject matter experts.

D

uh Now we even provide one-on-one coaching for people who are less confident and give people opportunities to go through multiple rounds or shadows and River Shadows for both service desk and incident response rotations. If they ask for some, people are confident some people are not so depending on the level of confidence. We give the opportunities.

A

Wonderful thanks, um yeah I've, never heard incidents reply uh described as a mystery Australia, but that definitely makes a lot of sense and there's a certain sense of satisfaction. Once you figure out what the root cause was, it's often hard to understand. What's going on with a kubernetes cluster when something does misbehave. For example, we've certainly experienced this where a misbehaving client can easily degrade the performance of the control plane.

A

What tools work well to get observability into what's happening with your cluster and to use any out of band tools to share context as changes are made by engineers at the company to try and understand what might have caused an issue with the cluster. Let's start with um yeah.

D

This is going to be a really interesting and long-winded one. um We use Prometheus plus graph for Matrix Vector Plus umu for logging, not just for kubernetes components.

D

Obviously, kubernetes exposes some of these things, but also for the extensions that we built, and we expect all the clients that run on our system to have some at least in-house clients to have some level of uh monitoring logging Etc distributed tracing is slowly being added to core communities, components right now, so because there was no good prior art, we haven't done a lot of tracing for for our kubernetes extensions applications, do have you have them and use them heavily, though.

D

But that said um we have a black box monitoring running against our clusters, all the time we call them as possessions and they run frequently depending on where they are running and what exact set of tests they're running.

D

um But despite all these, we have been in sales where none of them have given us full story or the full picture. um We had to get these speed, arms read through IP tables and then deep dive into the code itself to even mitigate.

D

um So there are all those challenges, but in addition, we are trying to train the team on all the tools that we think we might use during these incidents and the tools that we have used during the incident, so that everybody is familiar with the tooling, at least so that they can use it when it's required.

D

um In addition to that, we are also building a CD system, which we internally call as platforms: CD optimized for infrastructure change management.

D

um We are building visibility, features into the system to understand the changes that go into the system quickly across the board, as in not just the changes rolled out by my team, but all the height other high touch teams that are building operators against the system. So, hopefully that will give us the visibility. We need to understand the changes that go into the system, fascinating.

A

So there's a lot to it um is the sort of summary there, which is lots of different systems, lots of different ways to try and figure out what is going on with the cluster. um Let's go to Fabio um and talk about Netflix.

C

um I personally love the CLI and a lot of people might change you as well, so we use Cube CTL, GQ Cube CTX, a lot um we build Tony script said hop scripts. On top of it um also I found having we found in my team having an ability to blast SSH. Even though we hate we don't we don't love doing that, it's always very, very good to have that escape patch, um so blast SSH, parallel SSH is also important.

C

um We have so our observability truly mostly predates kubernetes, so we already had observability before we introduced kubernetes. So instead of using kubernetes native double quote um observability we try to plug whatever we have into kubernetes, and we mostly do that from a combination of writing our own exporters or using open Telemetry These Days Hotel collectors. Either we contribute our own custom collectors. We use what's available there.

C

uh Those collectors and slurpers are pulling all the operational data like logs tracing metrics and forwarding them to our in-house um observability tooling, which is a combination of Atlas, our Open Source Time series database and we um complement Atlas with indexing. uh We use a lot of elk elasticsearch kibana for indexing. Various things ranging from logs choose State transitions events from the Clusters, and we also complement that with a big big data style, um data, archival and coding capabilities.

C

Both of those indexing and big data is very valuable for us for ad hoc running as well. We often get to the case where we don't have the metrics that we needed to troubleshoot something it will add the metrics, but we also have the ability to look back at what happened. That's something valuable to us. We also. We also found it very, very helpful to write plugins to your Cube scheduler to add more tracing data, especially on scheduling decisions.

C

uh Our experience is that the Pod, the information about scheduling failures, that's in the Pod, it's good, but you have no history and it's often very hard to understand why things are not getting scheduled. So we also added a bunch of extra Telemetry there. Wonderful.

A

Thank you, Fabio. uh Let's go to Ramya.

E

um So we we have metrics, we scrape all the Prometheus endpoints that is available and then pop that to our uh pumped up to our metrics provider and then and then we have logging. We have a similar year case at all that collects all the logs from all the systems and make it available through kibana, um maybe have dashboards and during every incident, I figure out a new metric. That is super useful.

E

That was not previously there in the dashboard or that's what can have only like 50 or 60 metrics I'll be on that it becomes unreadable. But after every incident, I'm like oh, this metric is super useful I, keep adding them to the dashboard um so that the same goes with scheduling, scheduler control plan and kubernetes.

B

Api server kubernetes APS over being.

E

A huge metrics Matrix that is given out by kubernetes API servers a lot and uh we pick and choose only ones that matter. uh Events are super critical for us. Why did airport get killed? Why did apart get? uh uh Why did a pawn get evicted? Why did a scheduling failure all that gets complex metrics as events and we looked through I looked through events law event, logs all the time to figure out why scheduling failed or why upon got killed or why we rotated a node and stuff like that.

E

um So events are super super useful uh during large scale incidents.

E

uh We hope to have everything scripted out and have around work for, like you know, daily nodes or rotating uh rotating all deployments and stuff like that, but I invariably pick up keep signing your like my own for while loop during incidents to handle handle the because there's slight variation I only want to filter out all these deployments that have this characteristic and then do something to that second deployment, so I literally go back to Shell and Michelle and uh try to change the state of the system to handle incidents.

E

So um yeah that's been our experience. Handling incidents and.

A

Thanks very much actually tell us about the lift yeah.

B

So we also have an in-house stats pipeline that is uh collecting up and aggregating um in cluster, for media sets similar thing for logging, where that's collected off and shipped to Central Cabana. uh We do have uh and rely a lot on dashboards and alarms and as part of our weekly on-call um handoff process, there's tuning um and back testing that happens to ensure that these alarms are.

E

B

And not just being noisy, we also rely on an in-house Canary service, it's kind of similar to the assertions things just running into the tests continuously covering functionality that is essential for operating web services and painting us if it cannot complete it for operational, tooling, Lyft has open source the project called clutch that exposes sort of these like critical incident response actions such as coordinating a node debugging or like pulling debug logs from Envoy or terminating pods Etc in a web UI.

B

So we rely on that to respond to things and also enable others to kind of self-help like a service owner can Court a node if they, you know notice, it's an issue before the compute on-call is, is available um and that way like we can improve that sort of like time to mitigation and then in terms of how we um like handle misbehaving clients we do try to, as I mentioned earlier, limit the usage of operators and then also isolate these uh operators and Charities onto clusters, where their blast radius is limited.

B

uh One thing that we've built that we find very useful is we build a large like collection of these? Really small operational tools or like self-healing assistance tools for Kunis, because a lot of times, we'll notice that kubernetes will kind of get itself into this terminal state that isn't really what is expected by the service owner. For example, like uh you, might get a fail, create pod, sandbox and, like that's just the end for kubernetes, and it just leaves it there and we'll never try which isn't really what the service owner wants.

B

They just want their workload to run somewhere, and so we have like a little controller that, like will watch for things like this and just like terminate the thing according to node Etc, um so that this will just go somewhere else schedule and be successful, like the service owner expects uh when we think about these little tools and how we build them. One of the principles that we follow is Simplicity.

A

B

Are like uh things that we want to prevent or help respond to incidents. We want, don't want, like any incidents with these tools themselves, and so, if you build like a giant rub Goldberg machine, that's like managing and trying to do all the magic on your infrastructure, you're like going to get into trouble fairly easily um and like if something goes wrong with these things, it'll be really hard to figure out what went wrong.

B

uh So we try to make these things as simple as possible where these services, like the thing that goes and kills the pods and sandbox there is that's a separate tool. The thing that coordinates cordon's note that's a separate tool, so they're these really basic, simple controllers, um but they should be dead, simple and then just work to make things aggregate better. Hopefully, all.

A

Right, wonderful! Well, thank you all for sharing today. It's always really interesting to see. What's worked for everyone, what hasn't um so that we can all learn from each other, I've suddenly learned a few things today. um Thank you all for your time today and I wish you the quietest of on-call weeks going forwards thanks. Everyone.