Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: "Extend All The Things!": Cloud Provider Edition - Joe Betz, Google

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

"Extend All The Things!": Cloud Provider Edition - Joe Betz, Google

Kubernetes integrates with a ton of Cloud Provider platforms, and as part of the "Cloud Provider Extraction" effort, all platform integration code in the main Kubernetes code base is being moved out of into separate repos. To make this possible, extensibility points have been introduced that create interesting ways to extend Kubernetes, including Cloud Controller Managers, Credential Provider Extensions and the Konnectivity server and agent.

Learn the details from an engineer who helped bring the CRD and webhook extensibility mechanisms to GA, and who leads the adoption of extracted Cloud Providers at Google. How do these new extensibility points work? Why are they important? What can you do with them? We will discuss each extension point in detail, covering best practices learned as we migrated cloud providers to them.

The Cloud Provider abstraction delivers cross platform portability. We’ll close with details on how you (attn: tooling authors, cloud operators) can get involved with the community effort martialed by SIG Cloud Provider.

A

Welcome, my name is joe batz and today I'm going to be talking about cloud provider integrations with kubernetes and the extensibility features around that.

A

A couple things about myself: so I'm a std maintainer I've been working at google. I've been involved in kubernetes for about four years, I'm mostly involved in api machinery.

A

I've contributed to bringing custom resource definitions to ga bringing admission web hooks to ga and, more recently, developing the server side apply feature which is slated to go to ga in the upcoming 1.22 release.

A

More recently, I've gotten involved in sig cloud provider, and I'm going to be talking about that quite a bit. Today.

A

I've got a lot to cover. I'm still going to start by talking a little bit about the cloud provider extraction project, which is a fairly large initiative in the kubernetes community.

A

I'm then going to switch gears and focus on kubernetes extensibility features in general and then start to kind of work. Our way down to the more cloud provider. Specific extensibility features from there we'll dig into how those features can be used to build cloud provider integrations and help move the cloud provider extraction project forward.

A

I'll then finish up by talking a little bit about how this makes kubernetes better both for the kubernetes developers and for the ecosystem at large. So let's get started. Let's start by talking about the cloud provider extraction project. This is a project, that's led by sig cloud provider and it really comes from their core mission statement, which includes the goal of evolving the kubernetes ecosystem in a way that is neutral to all cloud providers to get a sense of what that means.

A

Let's have a look at the code, so on the left here we have a list of the cloud providers that are in the main kubernetes source tree. These cloud providers are uh directly combi compiled into the main kubernetes binaries um and are deeply integrated with the kubernetes code base on the right. We have the out of tree cloud providers. These are cloud providers that are exist in their own separate source. Repos are built out of their own binaries and only interact with the main kubernetes binaries through extension points.

A

You might have already noticed that there's some overlap like some of these cloud dividers are in both right entry and out of tree. There's a reason for this there's currently a migration going on to move all entry cloud providers out of tree. So these are the source and destinations of that migration.

A

When this migration completes, we will be able to delete all of this code out of the main kubernetes code base.

A

I haven't listed all of the kubernetes cloud provider implementations here. There are many, many other third-party bug provider implementations that are that are um really well built. um I have just kind of limited my list here to the ones that exist in the kubernetes community-owned source repositories.

A

um Let's talk a little bit about the benefits of this, so uh one benefit's kind of obviously we're gonna delete a lot of code out of kubernetes and that's gonna result in a you know, a more stable and maintainable core.

A

A lot of that deletion is actually going to be dependencies. The cloud provider implementations pull in a lot of complex dependencies and we're going to be able to take all of that out of kubernetes.

A

That's going to result, of course, in easier development, but it's also going to result in nice. Things like, like smaller binaries.

A

Of course. This also helps us achieve that goal of having like a cloud provider neutral ecosystem, and it does it in a nice way, because what we're doing is we're up leveling the extensibility of kubernetes, so that anything that the entry pro providers can do today will be available to all cloud providers.

A

Lastly, this is a big benefit for developers in general. If you are developing cloud provider, integrations you're going to be doing that your own source repository you're, going to have your own release process with your own cadence um you're, not going to be locked into the main kubernetes release process, which is fairly infrequent and we're seeing the benefits of of this um happening right. We are seeing increased activity and extensibility point development, we're seeing that increased activity around cloud controller um manager, infrastructure right.

A

The ability to build a cloud controller is getting easier because we're up leveling that as part of the migration and all third-party cloud providers, will benefit from that infrastructure as well I'll talk about that more, so we kind of get towards the end of the talk.

A

This project has been going on for over two years in open source. Just recently, we hit one key milestone, which is in the 1.21 release. A moratorium was placed that restricts any feature development from happening in tree.

A

That's a pretty strong incentive for these migrations to get completed and then in the 1.24 release, which should be roughly a year. From now all of the entry cloud provider code is slated to be deleted, which kind of brings this whole migration kind of to a close.

A

I'm going to switch topics now we're going to talk a little bit about kubernetes extensibility, um just to kind of warm up we're going to talk about some of the core extensibility features of kubernetes, we'll dive into some more advanced ones, and then we're really going to get down into the kind of cloud provider, specific capabilities um that are available. We're going to talk a lot about things that you would only really see if you were a vendor or cloud provider, adding functionality to kubernetes.

A

So, let's get started. um This almost goes without saying, but kubrick controllers really are the original kubernetes extensibility feature um the the idea in kubernetes that you have this central component in your control plane that doesn't actually have any logic in it and all of the logic is pushed out into processes that communicate to it via a common api.

A

That's a very kubernetes specific idea and it's really shaped the kubernetes api. The kubernetes api is a full api. If you can use that api, you can do anything that can be done in kubernetes, because everything that we do in kubernetes is done through it right. Even things like scheduling of pods and garbage collection, those are all done through the api.

A

This means it's a really powerful extension point. If you write your own controller, you can do pretty much anything that can be done and it's the way that you set up controllers is really well thought out too right. You can run a controller right there in the cluster. You can just tell kubernetes to run your controller and you just provide it with the container image. We'll start it up for you and manage it, you can also choose to run into the control plane. If you want to manage it that way, there's a lot of flexibility.

A

um The kind of the complement to controllers in some ways is custom resource definitions. um I really like that phrase that I hear a lot, which is that kubernetes is all about the api.

A

It's a really powerful, complete api and the idea that you can take and add more first class objects to. It is really powerful and that's what crds allow you to do and when you take and combine a crd with controller, you get things like the operator pattern right and that's a huge part of what the kubernetes ecosystem is that you can customize kubernetes to do almost anything and let's look at the mechanism you use so for a custom resource definition.

A

What you do is, you add an object to the api that says what your new resource type is going to be right and you you specify the format of it and how it works, and then you can just start creating those objects.

A

We're going to see these kind of patterns as we dig deeper into extensibility features of kubernetes. So, let's, let's go layer deeper admission web hooks, um so dynamic admission control is kind of the idea that you can intercept a request come into the api server and you could do whatever you want. You could reject it. You could modify it things like that.

A

Pretty powerful concept, lots of different applications to it. You can add security checks. um You could add additional validation to a crd.

A

You could make it so that every time a certain kind of pod is created, there's some side card. That's added to it, there's a lot of different things you could do with this, but it's more advanced and there's more sharp edges around it right. Each web hook is part of that critical, serving path of a request that gets the api server with. That comes a lot of responsibility right. If that, if your web hook fails what's going to happen, you have to choose.

A

If you choose to have your website fail, open, you're, saying that if my web hook, if the process I'm serving my web hooks from can't be reached by the api server, then we should just accept the request in which means my web hook wasn't run.

A

If I fail close, then what I'm telling the api server is that I, if my web hook process can't be reached that the whole request should fail.

A

Well, that's good in terms of like security, we're not introducing a security hole, but we are potentially causing the entire control plane to go unavail available if my web hook becomes unavailable. So there's a lot of responsibility there in web hook. Development also web hooks are part of that critical serving path, and so their latency really matters.

A

If you add a bunch of mutating web hooks that are slow, you're gonna add a lot of latency. In fact, you can reach a point where the cube api server will time out web hook. um It's it's web hook, handling requests and then it will fail. um So you have to you, have to keep your web hooks fast.

A

Adding a web hook is a relatively well-defined process and it's kind of like adding a controller right. You can just run it in the cluster using existing kubernetes constructs and then just um register with the kube api server, where it is, and it will talk to it and that's really easy to do too right. You just add a kubernetes object that explains your web hook and you're all set.

A

So this kind of fault, you're kind of seeing some patterns here, right, running, writing, binaries as dedicated processes, and then configuring how you communicate with them as a way of doing extensions.

A

The next extension I wanted to talk about is a little less commonly seen and it's called the aggregation api server pattern. um So what you do here is you tell the api server that for some api endpoints like for some url path, um don't actually serve it from the api server, but instead forward the request to some other system, and you get to decide where that system is, and you can control exactly how it works.

A

You can have it still act as though it is an api server and have it use. um You know, kubernetes object formats or you could have it do something completely different. It's entirely up to you, it's a really powerful extension point and uh uh can do a lot of different things.

A

um The way you use it again fairly similar right, you can just run, run your process either in in cluster or in the control plane, and then you configure the api server to tell it where, where your extension server is.

A

The next couple extension points we're going to talk about a little more low level. So typically, you would be a vendor or a cloud provider developer. That is adding your system to kubernetes.

A

The first one is the container storage interface. um So this allows you to extend kubernetes to support a vendor cloud provider, specific volume, implementation and the way that this one works is that you need to add something called the device plug-in binary to every node and then the kubla's going to talk to that. Well, one way you could do that is through a daemon set. You could use a daemon set and you will get a container running on every node and this works just fine.

A

So you could deploy your um your volume support onto the cluster that way and then what the way that this extension point works is that your device plug-in when it starts it's going to register itself with the kubelet by sending a request to a socket at a well-defined location and then once that happens, it's going to tell the kubelet where its socket is and the kubelet is going to then start making requests to it.

A

For container related operations, okay, sorry volume related operations on its socket, and so now you have a communication flow and everything everything's all hooked together.

A

The container network, interface or cni is somewhat similar here again we're doing we're, adding a process to the node this one, though you have to configure at kubelet startup, so you provide flags to the kubelet, telling it that it's going to be using cni and where the config for that is and where the binary is and then what happens is when the kubelet needs to set up some networking.

A

It's going to exec this binary as a sub process and then communicate through it via standard in and standard out.

A

This works pretty well because, typically, you don't need to run this process for very long like it might configure some ip tables and then shut down and you're done, um and so it's it tends not to be running very often and so kind of executing as a sub process ends up being fairly fairly low cost.

A

The the last of these these container interfaces is the container runtime interface.

A

This was introduced back when docker was still the only way that kubernetes could run containers and it was introduced as a way of allowing alternate container run times this one. The way it works is you provide flags to the kubelet on startup, telling it what sockets to communicate to for container runtime and for image service management and what the kubelet then does is just send requests to those sockets.

A

It's expecting that service to be running when the curb kublet's running, so you cannot run this service with a daemon set. You really do need to um set this up. The same way that the kubelet is set up right.

A

Part of the problem here is: if you try to use a daemon set well, what a daemon set's going to do is it's going to pull an image to run a process, and then it's going to start that container.

A

Well, what you're trying to set up here is the infrastructure to pull images and run containers. So that's a bit of a chicken and egg problem. So typically here you know if, if you are, if you provision a node by using system d to start the kubelet, you would probably also use systemd to start that cri service, that you need.

A

So now, let's talk about some of the new extensibility features that are available, we're going to talk about kind of three different areas: we're going to talk about networking between the control plane and the cluster, we're going to talk about container image credentials and we're also going to talk about the more general problem of how do you run cloud specific controllers, so first networking, so one limitation with kubernetes is that it is not secure to run the control plane and have it communicate to a cluster across an untrusted network.

A

That is, that is not something that is securely supported by kubernetes and um it's generally problematic because, like you're gonna have to somehow what open ports to every node in your cluster on an untrusted network so that the control plane can communicate to it. Even just managing those firewall rules would be a real, a real nightmare.

A

um So what um this extension feature is it's called the api ing server. Egress is the way that you can have the api server talk to a proxy which then is responsible for talking to your cluster, and so you can basically introduce a you know. A single communication channel from your control plane to a cluster, and then you can go ahead and secure that as you need and I'll talk about like a de facto implementation.

A

This is pretty good and the way that that extension feature works is that when you start the kube api server, you provide a configuration file and that configuration file is a communities object that says what the redirect rules are. So you can both redirect traffic to the cluster so that the control plane can talk to all the nodes. You can also redirect traffic to other components of the master, so, for example, with the kuba yes server needs to talk to etcd. You can proxy that communication as well.

A

um There is a pretty good implementation of this extension point called the connectivity server. It uses um mtls over tcp for everything. um So it's a pretty standard. It's easy to set up a single firewall rule. You get that one connection, it's bi-directional, so it you're both your control, plane talking to your nodes and your nodes, talking back to your control plane all can go through that same system um and um all everything's everything's along that whole communication chain.

A

You have mtls and you know, site inserts all the way across.

A

For container image credentials, this is an interesting problem. So when you, if you're building a cloud provider, um typically your your developers might have a container registry which might be private right. So, for example, if you're running um you might have an account in the cloud provider as part of that account, you have your own container registry, which is private to you, and you can upload images to it and do everything you want to do um it.

A

Would you also need the credentials to that container registry to pull the images onto your nodes so to have a really nice turnkey cloud provider, implementation of kubernetes that should probably all be integrated together and that's what this extension point allows you to do. You can tell the kubelet um where it can load credential providers from and then you can provide a binary that does whatever you need.

A

This uses that exact subprocess pattern again, where the kubelet, whenever it needs credentials, it's going to start a sub process and then talk to it through standard and standard out when it's done it will shut it down. It's not that frequent. It's going to need two new credentials. So um this works out pretty well, you configure it by telling it kind of a list of different images and where it looks up credentials for each of those and then once you've got this all set up. You should be good.

A

A good example of this is like, for example, in gcp. um Every.

A

Every image um every vm has a metadata endpoint and so what you, what and so those can be loaded up with credentials and then the credential provider can just grab the credentials from that metadata endpoint and provide it back to kubernetes um in a different environment. You might get credentials from somewhere else. There might be some service that you load them from or some other way of looking them up.

A

You can do any of those with this extension point.

A

Finally, the last thing we're going to talk about is cloud provider, specific controllers, so in the main kubernetes code base in the entry implementation, there's a bunch of controllers that do cloud provider specific stuff that includes node, related controllers, so creating new, creating nodes and deleting nodes often means creating or deleting vms.

A

um It can involve volumes because, there's a lot of like cloud specific storage systems involved, it can involve routes and services where load balancers are involved or, lastly, it can involve like ip address management where, like ipad ranges, need to be allocated um or provisioned for various things.

A

All of those controllers um in the entry implementation were activated. When you set flags on the coop controller manager, you would say which, which cloud provider you were using and then you would try some additional config and those the right set of controllers would be run um in the out of tree architecture. The coupe controller monitor is not going to have anything cloud specific in it. So when you run it, these controllers will not be run instead.

A

You're going to every cloud provider is going to have its own cloud controller manager that it provides us a separate binary when you start that you'll provide a bunch of cloud provider, specific flags and it's going to be responsible for running those controllers.

A

This is a pretty big change right right now, if you're running any of those intrigue, cloud providers you're only running one controller manager by the end of this migration, there's going to be two controller managers running.

A

So in order to achieve that, there's been a couple things that's happened. One is that there's been a bunch of improvements around controller management manager, infrastructure.

A

This makes it easier to define and author new controller managers and wire in all the flags that you need um and also, very importantly, it's added ha support for migrating controllers.

A

So if you are running a kubernetes cluster, that is mission critical, you may be using aha control plane, which means you're running multiple replicas of the of every component: the control plane, the api server, your scheduler, your um your controller manager, everything you're going to run um probably like in triplicate, and um if one of them goes down your your your control plane is still up or if you need to do a rolling upgrade, you can upgrade the entire control plane without any downtime.

A

We need a way to do the cloud controller migration, moving those controllers, those cloud specific controllers from the intri architecture, to the out of katrina architecture without any downtime.

A

This is a little tricky to do and I'm not going to get into all the details. But it was a really interesting um there's a really interesting solution um that is being rolled out where um we use multiple leader election leases um and or blocks sorry, multiple leader election locks. So basically, what happens is the coupe controller manager claims two locks, one for all cloud provider, agnostic controllers and then another for the cloud provider, specific controllers and then over time? What will happen is in this.

A

The subsequent release, the cloud controller manager will be, will start to be run and it will try and claim just the lease for the cloud-specific controllers and then, once the um you release a version of the cube controller manager that doesn't have cloud controllers in it anymore, then only the cloud controller managers can be able to claim that lease, and so you get this kind of series of operations that happen, where you hand off ownership of those controllers from one system to another.

A

But at any given point in time you still have the same guarantees around controller management, which is only one controller, is being run actively at any given point in time, but there are still backups that can take over if that one shuts down um the last thing, I'll mention about controllers is that if you are doing cloud um provider development, um something like coupe builder, is a really good option.

A

We're doing a lot of improvements around the core controller manager infrastructure, but a system like coupe builder, if you are building a lot of new controllers, is a really good way to do that.

A

Our team has quite a few engineers that work very closely with coop builder and it's a really solid system.

A

That's it, though. The thing I'll kind of end with for everyone here is that when we get to a point where we've really moved all of this kind of cloud specific and uh code out of kubernetes, we really have this cloudless kubernetes core.

A

It has less code in it less cloud specific stuff in it, but more extension points it's a more versatile, more tightly knit system, um and I really think that is going to help with the longevity and health of the kubernetes code base.

A

I think it's going to start to build an increasingly vibrant ecosystem. I'm really excited to see like what other extensibility points people find that are needed. If you do find that you have a gap, please come talk to the cloud provider sig and we're very interested in hearing from you to get involved. You can come to our slack channel or our mailing list and we meet regularly every wednesday at 1pm pacific time. There's a lot more details on the link here to our sig page.

A

Thank you and I will open it for questions.