Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Custom Controllers to the Rescue: Tailoring API Objects to Your... - Stephen Chan & Davi Arnaut

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Custom Controllers to the Rescue: Tailoring API Objects to Your Needs - Stephen Chan & Davi Arnaut, Airbnb

External controllers and admission controllers are a great way to extend Kubernetes and enforce cluster policies that are particular to your needs. In this talk, we’ll showcase how to use controllers backed by real examples used in production at Airbnb. We’ll show how controllers can help the scheduler spread a deployment’s pods across zones more evenly, enforce maximum pod age, and taint nodes based on conditions set by node-problem-detector. We’ll also show how we use admission controllers to enforce naming conventions and security constraints such as container image source on API objects submitted to a cluster.

To learn more: https://sched.co/GrUC

A

Welcome everyone: my name is daddy. This is tava Chan. We are from Airbnb and we're gonna be talking about custom controllers and how we make use of them in our infrastructure. For those. Don't that don't know about Airbnb or then be its online marketplace for homes and experiences you can find homes, apartments and all that sort of stuff on Airbnb, just a quick plug. We are hiring.

A

We are hiring for a lot of positions, infrastructure, side, the ability, payments, mobile and it's a special time should join our ability because we're basically in the process of changing how transforming how our entire infrastructure works, and we have offices in San, Francisco, Portland Seattle Beijing. So you can pretty much work from a lot of different places. So one of the first things are going to talk about this.

A

Why do we use kubernetes, so air B&B until a few years ago, used to be a very few monolith caps and like by mulleted caps, I, really talking about a single app with millions of lines of Ruby called Rails based, and we had them similar small, some small other services, and we are starting the process of breaking that all apart and moving into a services, oriented architecture and we've been doing this for some time and we already have a few hundred services and one big thing with those services that we used to be a chef based so which it's service would have its own like recipes and everything chef and they would run on their own ec2 host and that's very, very expensive, because you end up with a bunch of machines and we didn't have a processor out to scale those someone had to go manually, launching the machines or things or if it was taking.

A

Let's say different. There was a peak used to have the same number of machines. We didn't have anything like to automatically placed hosts that died, so we decided as part of the service-oriented migration. So, let's start moving those services also into kubernetes Melanie early during the keynote talked about a little bit. How we deploy things on kubernetes, but one of the reasons we went with kubernetes I would say that one our main driving was actually cost. Like I said we had one service, a very serious.

A

It means we couldn't pack as many services as we wanted into a single host, so with kerbin ares we can probably more better skilled those services. Let's say most services are gonna, probably going to be using like one choose, abused and a few gigs of memory, so we can basically have a larger host and pack more bang for the buck. On those also another thing that kubernetes providers is like self feelings of my node. I he automatically migrates how the pods, when you know, then we don't have to worry about that.

A

It's basically unknown event and we can actually even have like all the scaling for certain services so that they can scale up and down with traffic. So let's talk about what is actually a customer. Truly, a cousin controller is just a component that is watching objects and looking at what is the state of those objects? What is the current state of the cluster and maybe doing modifications to that state so that it reaches a desired state? So kubernetes is a declarative day's assistance.

A

So you declare what is the goal state and then there is a lot of things most of those are controllers that are watching the state of the concent trying to move this state should that desired state.

A

So why do we use console controllers at their B&B? Are mine used for coastal controllers action, integrating coronaries with external assistance, for example? One controller that currently it's in criminality is that, for example, if you have like an EOB, you want to have like a load balancer Bursar's. There is a controller somewhere that basically goes and talks to AWS and provisions that control it for you, we use that we extend that concept to more things.

A

We use it for like 20 greatest services cover E or for exemption provision to provision in the bless resources, and things like that, and you can even use it to this- can make a lot of requirements about how your objects should look like and you can actually use. It should do a lot of validations. You can use it to validate your objects, that they have the required views that they've followed best practices and another another particular type of controller that we use.

A

A lot is an admission control admission control is basically it's a controller that intercepts requests after they had been authenticated and authorized ated so like when you do a crib CTL command, you're talking to the API server, so that the mission controller gets called for the API server, but only after it has been authenticated and authorized it. So the request is basically identical, so you have access like username and things like that. Why do we use admission controls?

A

We said mission controllers for actually a few different things, but the main one is actually for us to require a standardization of our objects and when I talk about objects, when I'm talking about things like even a pod object, I'm making sure that, for example, the pod object has all the fields that are required to be deployed within our cluster. We can all see we can and initial controls they have to.

A

They have validating and mutiny controllers, so you can actually, for example, validate monic, and you just just to check things in your pods back and make sure that I think there is a part. Is there or also the mutating one? You can actually use it to modify things so, for example, you can use it to inject sidecars in a pod and that sort of stuff. Another thing we use it for is that we can use it to secure our cluster.

A

For example, we can make sure that if someone is trying to deploy like there was a keynote that talked about this, someone could actually try to deploy something from the internet to create service accounts, and things like that. So we use this, we use the validate across the controller should prevent those things from happening. So we want people want.

A

Our engineers should only be able to deploy like blast applications, so we use admission controllers to check for those things, so just a quick overview of that mission, controller that Mission Control is operating before the object actually gets persisted or created like before. You do any change to the changes, so the objects are are persisted, so that's the part where and the mission controller works, and then we have the custom controllers they're, basically operating after those steps and I'm gonna hand over to Steven to talk about our specific use. Cases for controllers thanks.

B

Duffy yeah so now that we've covered the high-level like concepts and motivations we'll go through some of the existing use cases that we got from writing admission controllers and custom controllers. So, let's go through how a year's admission controller. First of all, as Debbie mentioned, you can use it for validation and mutation. However, we currently only use it for validation. We find that it's preferable to keep all the transformation of objects before apply time using cube gen we.

B

What are we validate it for we validate the presence of standard annotations on all pods that are created in cluster, so these have links to, for example, our CIC DUI, so you can see when a particular pod was or service was deployed last and who deployed it.

B

You have project owner contact information, so it's got a database of mapping services to team owners and then finally, we've got links to project repos and get Shaw, so you can view the source code that was corresponds to the creation of pod and because we have our deploy and configuration alongside our code. It's easy to view everything at once. I should also mention that, because of the project owner contact information annotation, we can use this not only for debugging incidents but also sending out automated alerts.

B

So we can alert the owners of that particular service if their part is consuming more than their requested amount of CPU or has experience in Health memory.

B

We also use admission controllers to secure and create enforce best practices on a cluster. So we enforce that all the container images on pods come from our single trusted registry source. We also prevent configurations that aren't meant to run on our clusters. So, for example, we don't allow the balancer type services right now, because our clusters are in internal facing. We don't want them to create external endpoints.

B

Finally, we just use it as a prevention of for guns, such as having developers deployed their development services to production clusters yeah. So we've gone through admission controllers. Now we're going to talk about some custom controllers that we build so first off, we've got any service discovery controller. So why did we build this? We built this to integrate kubernetes with our cluster external systems. In this case, we built it to integrate with smart stack, which is our existing service discovery framework, so it smart stack in case you're, not familiar with it.

B

It's an open-source service discovery framework. It's got a few components. It's got the zookeeper, which is a distributed, key-value store on the client side. You've got synapse, which is polling zookeeper for for backends for particular service. It uses that list to configure a proxy which does actual load balancing when the client service makes requests on the server side. You've got nerve, which continually health checks the server side and ensures that if the service is healthy, its back-end is built one zookeeper and if it's unhealthy, that's remove from the zookeeper.

B

So this is what it looks like before kubernetes. So what does it look like inside of kubernetes? So we need a few more pieces of information which we get from the API server. We implement this as a controller sidecar within each pod. So what that does? Is it's reading the note port service information previously in smart stack, we expected that ports were relatively statically allocated in kubernetes when you create an airport service that port is randomly allocated.

B

So you need to fetch that information, and you also need the current host that the pod is running on, and so you see here, the controller sidecar has replaced the role of nerve. It still does health checks to determine the readiness of the serving pod and then removes it or adds it from the zookeeper accordingly.

B

So to summarize, the objects that the this controller watches our services pods and notes the goal state is to add or remove to zqa the zookeeper based on the health status of the pod.

B

Next up, we've got what we call the deployment pruner controller, and so why did we build this one? We wanted to add additional behavior on top of deployments, so what we want to do is enforce max applaude lifespan, while respecting max unavailable count. So on all our clusters, we enforce that pods are not around for more than 14 days, but if a service has been running 14 days without any restart of pods, we don't want to immediately take down that service.

B

So what it does is it watches the appointments, pods and notes, and then it will continuously delete pods as they've reached expiration and there's also a natural extension point for if we want to enforce any other policy around like pot expiry or if we want to add, like new new criteria to admission control, but the pods are already running, we can delete those as well.

B

Third, we've got the no problem controller again we're trying to layer behavior on top of existing API objects. So what does the no problem? Controller do well, primarily watches nodes and it's responsible for providing another layer of self-healing on top of our clusters, so it works in antennae with the no problem detector, which is an Austrian criminales project. What the no problem detector does is it health checks, the underlying minion nodes and then sets conditions to the nodes according to the results of the help checks, what it doesn't do is provide any remediating action.

B

So it's still up to you to add that additional top, so no problem controller will check those conditions and add taints to nodes that have undesirable conditions, and so we can use this for enforcing things like the presence of data dog or cubed. I am daemon set pods. Alright. Finally, we've been experimenting with using meta controller as a general purpose pattern for resilient architectures, so inner keynote, my colleague Melanie talked about using meta controller for creating AWS custom resource definitions and then replicating those to provision.

B

I am roles or policies. We also use it for things like alerts, so you can create an alerts CRD and then the alerts controller will ensure that the alert definitions are created in our external monitoring system.

B

Okay, now that we've gone through some practical and some of the controllers that we use, we can go through practical controller tips when you're writing your own controllers, so tip number one, always autoscale your controllers, as your cluster size grows. If you're using the go client your you're going to be maintaining a local cache of your objects, and so it's easy for that to creep up on memory and CPU usage as you expand the cluster.

B

So in addition, you can also use labels and selectors to optimize your data fetch, and there are also a few like options for doing this. Auto-Scaling can use vertical pot, autoscaler or pot nanny, which is sidecar, but one important thing here to note is that part: nanny itself is kind of controller. It's watching the nodes of the cluster, and when we had a case where we were using a hipster and running pod nanny as a sidecar, we found that hipster was unavailable.

B

We thought that squared keeps to his auto scaling, but actually pod Nana, pod nanny was out of memory at that point, so that made the service unavailable tip number two definitely use a go client, so kubernetes supports several different languages for interacting with API, but the go client you get. The latest features from extreme yet informers, which are basically web framework for calling your calling your controller indexers, which are the local cache of objects.

B

We got leader election for high availability event, spam filter, so one thing to note with larger clusters is that high numbers of events can sometimes overwhelm and CDs. This prevents you from doing that. If your controllers, omitting events, you also have more client sets so that you can easily test your controller by submitting fake objects that you've created. You can also import from other libraries, such as API machinery or like API server, depending on whatever utility functions you might need and the bottom of this slide.

B

There are a couple links that I found helpful when first learning about custom controllers how to write them in particular. The first link has a really helpful diagram for understanding the interaction between the Informer's, indexers and so on in deco client tip number three: there's really no silver bullet. What I mean by that is that controllers are great.

B

Kubernetes is great. They only bring. You part, waited the solution for spreading the control loop pattern to a large number of people. You can lower the barrier with tools like meta controller, but at the end of the day, you still have to have a deep understanding of the underlying system be unafraid to dive into the source code, if necessary, or and be careful with your experiments. So with that, thank you. We hope you'll enjoy.

B

Writing your own custom controllers and want to find your admission controllers.