Kubernetes Cloud Provider Special Interest Group, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cloud Provider: Portable K8s Across all Clouds, Roadmap and Updates - Nick Turner & Steve Wong

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

SIG Cloud Provider: Portable K8s Across all Clouds, Roadmap and Updates - Nick Turner, Amazon & Steve Wong, VMware

Cloud Provider code allows Kubernetes to run on top of different platforms, with an implementation for each. The agenda will include: An overall status report on removing the cloud provider code from the main Kubernetes repository to “out of tree repositories; “Lightning talks” for individual cloud providers, reporting efforts, accomplishments, and roadmap for features and getting "out-of-tree". We’ll also discuss the plans to handle cloud provider migration - including interesting topics like building and migrating to cloud controller managers, and kubelet image credential providers. The goal of SIG Cloud Provider is to promote a vendor-neutral ecosystem for our community. We will close with details on how you can get involved with the SIG as either a cloud infrastructure supporter, a K8s distribution author, or a K8s user.

A

Welcome to kubecon 2022 in valencia. This session is about the current state of the kubernetes cloud provider. I'm steve wong of vmware and I'm joined by nick turner of amazon, we'll quickly cover what the cloud provider does. Just in case, we have some audience members who are new to kubernetes, then move on to general project status, followed by lightning talks from individual cloud provider, implementations.

A

Finally, we'll wrap up with futures and coverage of how to join the cloud provider sigs community. The cloud provider is the mechanism that allows kubernetes applications to be portable across various public and on-prem clouds. The goal is an experience where any well-written app can run anywhere and, for the most part, can't even tell where it's running sig cloud provider works closely with a few other sigs to make this happen. For example, most of the storage related abstraction is managed by sig storage, with a little integration activity across the sigs being monitored by sig cloud provider.

A

In the early days, kubernetes was a monolith when it came to cloud providers. The kubernetes binary included a bunch of code for cloud specific cloud providers and it was bigger than it needed to be. This had a number of sub-optimal aspects listed here, for example, a user investigating startup logs on a google cloud might see entries saying could not find a aws ecb.

A

The old model slowed the time to future feature and patch delivery, and we've been moving to out of tree code for cloud providers for about two years now.

A

New deployments should be using auditry versions of the cloud providers now and if you're running a legacy version, you need to be planning for a migration, and nick is going to tell you more about that migration status so nick. Take it away.

B

Yeah, um so I'm going to talk a little bit about just kind of general status of cloud provider, migration and what components that involves um uh it's, it's actually kind of funny.

B

I just ran into lucas caldstrom, who was a very early member or even the genesis of cloud provider a couple hours ago, and we were talking and he actually started doing this work six years ago, and he was thinking like how long is this going to take and he in his pessimistic estimate, he was thinking about a year year and a half, and it ended up not being quite that short and there's a lot of reasons for that.

B

But it was more complex than people realized, and I think one problem was that there wasn't a lot of motivation for cloud providers to move out of tree because everything works in tree. Why should we go out a tree um but yeah? So um the first component that I want to talk about is the cube controller manager.

B

This is kind of what I think of when I think of the cloud provider in kubernetes. You have the control loops that act on. You know all of the objects and they also do things. You know making api calls to cloud apis, for example, managing instances, managing load, balancers routes, volumes, etc.

B

So, when we're talking about migration out of the cube controller manager, the node lifecycle, controller, the service controller and the route controllers are the primary controllers that we're referring to, but there's also the volume controllers which are going to be replaced by, for example, the csi drivers and when you pass cloud provider equals external to the cube controller manager, those all get disabled, and so, if you're running a cluster, that is, ha for example, and you don't want to have to uh experience downtime when you do this migration, then you're going to want to take advantage of the leader migration that we've built, which basically allows the cube controller manager and the component that replaces those cloud loops.

B

The cloud controller manager to use an additional leader, migration lock so that they can coordinate and not be. You won't have a situation where you have two leaders. You know two service controllers working at the same time, for example.

B

So that's something you might want to take advantage of. If you are, you know, if you have an aha cluster, if you're using a vendor, then the vendor's probably going to deal with that for you and you're not going to have to worry about it and, as I said, the cloud controller manager is this vendor-specific component that replaces those control loops in the cube controller manager, um but there's other components that are involved with this whole migration effort, the first being the api server.

B

So there's some lesser known, features that are considered part of the cloud provider, migration effort, one being ssh tunnels, which was something that really was only used by google and it doesn't involve cloud sdk. You know cloud provider, specific code, that's actually used. You know, there's no calls to the sdk there, but because it was specific to one cloud, we're kind of combining it with this effort and we've already extracted that from the api server and it's being replaced by the network proxy.

B

So if you are interested in that effort, take a look at the network proxy and the second piece in the api server is the persistent volume labeling admission controller, and this is a not super common use case. But it's when you need to create persistent volumes and you you don't have a pvc for them, but you still want the topology labels to be attached to those volume objects.

B

So this is an effort, that's actually still underway.

B

We are replacing that admission controller with a web hook that will be built into the ccm or, if the you know, if your distribution wants, you could run it separately, but idea being that you have a web hook. That replaces the admission controller there and then the next is cubelet. So cubelet also has a some cloud provider code that needs to get extracted.

B

There is node addressing functionality, so when cubelet starts up, it needs to figure out what addresses it needs to attach to the node object, so that communication can happen between the node object, the node and the api server, and so that is being replaced by a brand new controller.

B

That's being added to the cloud controller manager, which is the cloud node controller, not to be confused with the cloud node lifecycle controller or the cloud node ipam controller, which I'm not even going to talk about so node addressing is, is one piece in cubelet: that's important and needs to to move out. The next is.

B

There is actually some volume plug-in specific code in there, um so for I'm, only familiar with the aws stuff, but aws has some ebs mounting logic in there that uh needs to get out, and so that volume plug-in code is being replaced by csi migration.

B

You know the csi driver is going to do all of that work and, finally, the cubelet when it starts a pod, needs to pull the you know the docker image for that pod. um So there is a couple of plugins in there, for example, ecr gcr and maybe a few others that has the sdk logic that actually gets the credentials so that it can pull the the image and that's being replaced by something that we're calling the cubelet image credential provider, which is just going to be a binary that sits next to cubelet.

B

You have a configuration file you give to cubelet and then cubelet, just execs that binary to get the credential that it needs to pull the image.

B

And some random things to know so, for example, if you're doing this migration and you have uh and you you don't- want to use csi migration, but you do want to uh in general disable the cloud provider there's this little workaround, you can pass cloud provider equals external and you also need to pass external cloud volume plug-in to prevent those volume loops in the cube controller manager from being disabled.

B

So again, that's only when you want to do this without csi drivers enabled I wouldn't really recommend it. I would say just do everything at once and enable cs csi at the same time that you do your cloud provider, migration or even before, but that's an option. If you need it and another sort of interesting piece, is you may or may not be familiar with a feature in cubelet, which is the nodeip flag?

B

This is a flag that you can pass to cubelet, to give it a little bit more information about what ip address or you know, address that it needs to use and what happens is when you pass that flag cubelet will use it to filter down the addresses that it gets from the cloud provider when it calls and asks for the node addresses that it should be adding to the node object, and so this introduced a little wrinkle. um You know when uh the ccm is doing the same thing.

B

When you have that cloud, node controller, that's trying to figure out node addresses by calling you know what whatever cloud api it needs to do. Aws would be like ec2 or something to get those addresses. It didn't have that node ip to do that filtering, so there's actually an annotation that you can use, which is the equivalent when you're running ccm versus cubelet, which will allow you to have that same filtering logic, and there is a period of time during upgrade.

B

So there was a bug where cubelet and the cloud node controller both are trying to reconcile at the same time, and so we have a fix that will uh basically allow us to. We will always provide this annotation that allows this filtering to happen so that they agree, and there is a slightly better long-term fix that we're considering, which is to basically prevent them from ever uh try to do that reconciliation at the same time.

B

So that would be a ideal future fix. But currently, as long as they agree, then you know reconciliation uh is not gonna. You're not gonna have a flapping situation where addresses are going back and forth.

B

um So here are some slides from cloud providers that kind of go over the status of their migration process where they are etc. And did you want to do ibm.

A

B

A

A try um we've got these slides submitted by people who couldn't be here. You know obviously, kubernetes runs on a lot of clouds and we invited the uh people behind the cloud provider implementations for these clouds to give the community updates.

A

You can see the density here. I'm I'm just going to show these rather than read the slides to you. You could read them a lot quicker than I could pronounce them and we are publishing this deck, so you can download it later at the very end of this deck, you'll you'll get a url and a qr code where you can download the deck. So in this case this is so dense. I'm I'm going to assume you read this um they're covering interfaces.

A

Recent work and this one I can talk about and I'll do it very quickly. This is the vsphere cloud provider.

A

The summary is that the recent releases bumped up the level of support for recent to match recent versions of kubernetes and the other big news is that support for dual stack, which means running ipv4 and ipv6, simultaneously moved into alpha status, and we anticipate that it will move to ga pretty soon um azure uh penfe andy key couldn't be here: uh they're based in shanghai, so travel difficulties, obviously, but the stuff here.

A

The summary is that it's seems to be related to storage and networking, including once again dual stack, support, which I think is something that a lot of the cloud providers are either out with they're coming out with soon aws. I guess I'll give this back to you. Nick.

B

B

Yeah, so just really quick update for aws, we have stable releases of the cloud controller manager going back to uh 120 up through 124. Now we have a component called the aws load, balancer controller. This is where we're focusing our load balancing support.

B

So we have nlb support and alb support in the load, bouncer controller, and we recommend moving to to use it over the built-in uh service controller and that's just because we have a lot of uh people, maintaining it a lot of effort there. So you get better support and it supports the you know: the newer load, balancers, alb and nlb versus the uh the classic load. Bouncer, that's entry and so update about usage of ccm eks has we've migrated to use the ccm and 122.

B

um currently uh just the service and node lifecycle controllers and uh cops, also, I'm not sure if it's by default, but they provide you can enable the ccm in I believe, 122 plus and then csi migration is offered as add-ons in eks and cops to use the ebs csi driver.

B

Google, I guess I can do try to do google so.

B

Focus on google well, walter here is bragging about how many features are only tested on google gcp in entry. um So a lot of testing coverage there and that's actually something. You know that is, makes this whole process difficult. If we disable the cloud provider entry, then all of these tests start failing. So that's something that we need to work on.

B

We need to get these tests out of entry so that we can actually disable cloud providers and finish this process, but yeah so more gcp specific code in kk than other any other cloud provider mentions that the ssh channels have been removed and more more stuff to do.

B

Link to the reference implementation for gcp cloud provider and status and a little bit about what's left.

B

Okay, so just a couple notes on what is coming for sig cloud provider: we have these feature gates that will eventually flip to beta and when they do flip to beta cloud provider, stuff is just going to not work. It's going to be gone, it's going to be disabled, so you can always flip that back for a certain amount of time and re-enable them, but it's kind of a very strong message that this will be removed in the future.

B

So that's not going to be flipped in any of the immediately upcoming releases, because we have to finish some of these things like getting the you know finish the getting the stuff out of the api server as one piece, but look for that in the future. So that's eventually going to happen um and, uh as I mentioned before, um testing is uh is something that that we're thinking about so figuring out a plan. This is a proposal last known good, um which was, uh uh I think, we sort of in the last year.

B

We we discussed this, um but it's essentially around. How do we get those tests out of uh kk um and we want to bring those tests? We don't want to just get rid of them. We still want that coverage, so we want to move those tests to each cloud provider's.

B

You know ccm repo, but we still want those tests to matter. So the question is: are they release blocking or you know? How do we get that signal to the community so that we know that you know some change that was merged in kk breaks all the cloud providers?

B

So that's a proposal that uh is probably worth taking a look at and uh you know giving feedback on and some other random things um there's a clustered directory which walter has been trying to get rid of for a very long time.

B

So we'll see if he makes any progress on that in the next year and just in general, making all of the cloud providers work better right. We need, we all need to increase testing. We all need to. You know, put a lot of effort in to make sure this whole migration process goes smoothly.

B

And just some information about our sig when we meet, we have a couple bi-weekly meetings, there's the regular sig meeting, which is wednesdays at 9 00 a.m. Pacific time- and we also have an extraction focused meeting which is bi-weekly at 9 30 and that's on thursday, and they alternate so you have every other week. You have extraction and then regular sig meeting, there's the slack channel and video recordings, and I think that is yes.

A

I think we've got some time for qa, so please raise your hand, so we can get it on the audio for the remote audience and then, if you want to download this deck to get all those links, use the qr code or this url.

A

Anybody with a question.

A

Oh, let me turn it on there. You go.

A

C

Can we get the hand mike turn on?

C

I think now you can hear me right: okay, yeah, okay, so thank you for the talk, uh and this is, I think, an underestimated uh topic that many people don't look at and it's gonna cause some pain for some organizations, and the question that I have is mostly like around the cloud providers themselves and how does this migration is going to look like for customers of these cloud providers, because I know that some of them try to do the lift and shift on their own, and some of, like some others, try to push this responsibility of running this component to the customer which to me, as a customer, it looks like there was something that was working and it was managed before and now I need to manage it and like specifically, this is awsome balance controller and before it was integrated into the into the monitor's control plane.

C

Now you need to run there's a component uh and you have to maintain it basically.

B

Yeah um yeah, ideally you're right. It should be as easy as possible for the users so I'll start with just kind of general. uh Most things users are not going to see. You know a distribution is going to handle a lot of these things for you like the leader, migration is going to be, hopefully invisible. Unless something goes wrong. I guess you, depending on how much control you have over your nodes, you would see you know if you are providing uh you know.

B

If you build your own node uh images, machine images, then um you might have to do the work to uh put the cubelet image credential provider on the image right.

B

So there's some things that you might notice, but in general it should be pretty uh opaque.

B

I guess, um let's see so and and then the load bouncer controller um yeah, I'd like to have a better story with that, where the you know, someone who's setting up their own cluster doesn't have to worry quite so much about installing it as a separate component.

B

So we talked about different things like combining it with the ccm. That's still an option kind of turning it into a library or something adding it to the ccm.

B

I I like that idea, but we're not settled on it yet or just making. You know having like a really easy installation method that covers that, like, however, whatever it is, you know like if there's like a helm chart or something I don't know if helm is the best option for that, but something like that so uh yeah, I guess I can just say I recognize the problem there um does that.

C

Answer your question yeah I'm going to add on this, um and I think the problem is not with the new clusters, because new clusters are going to have the load balancer managed by the new aws balance controller, it's mostly about the ultra low bouncers, and how do you get them all outside of the entry management right?

C

Because for as far as I know, the ebscsi driver will provide a smooth migration path with these two feature: flags on the control api server, but for the aws advanced controller like basically, you need to migrate to a new load, balancer managed by aws load balance controller, which might be a pain.

C

If you don't have the correct mechanisms in the org to be able to do that shift transparently.

B

Right yeah, I think the idea is that you at this point you run both of them and and they both you know they shouldn't conflict um but yeah. If, if and when we do uh deprecate, I don't know that it would ever really be deprecated, because it's kind of tied to classic load balancers.

B

um But if and when we did, it would be. There would be a migration path for sure.

C

All right, thank you and another question. Sorry, um so do you plan? I saw these two uh flags that you have to completely disable the cloud provider. uh Do you plan, like? Are the cloud providers planning to expose these into their api so like as a customer? Again we have say we have migrated to the new um system.

C

uh We would like to completely get rid of the old one so that we can make sure that our customers do not create load balancers during the old, the whole system, for example, uh I'm not sure if it was clear you.

B

Got it? um No, they really shouldn't be exposed, um and in terms of I mean so, the for load balancers like we're really just talking about the entry.

B

You know service controller, that again speaking specifically about aws, but we're just talking about the entry service controller for the migration um and uh it shouldn't change right so like you're, still creating the same load balancers before and after the migration.

B

If you choose to install the aws specific controller, that's separate, and you can do that whenever you want all right.

C

A

We've got time for one more one or two more questions. If somebody's got one.

A

Okay, I'll take that as uh no, but you can track down. Nick and I are our names are in this deck. So if anything comes up and you can find us on the kubernetes slack in that cloud provider sig channel. So thank you for coming.