Kubernetes Cloud Provider Special Interest Group, 12 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cloud Provider Update - Michael McCune, Red Hat & Bridget Kromhout, Microsoft

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

SIG Cloud Provider Update - Michael McCune, Red Hat & Bridget Kromhout, Microsoft

Speakers: Michael McCune, Bridget Kromhout
Cloud Provider code allows Kubernetes to run on top of different platforms. Originally, support for all variations was built directly into a K8s release. This brought bloat and created an unnecessary dependency on cloud-provider specific code. This talk will be a status report on the effort to remove the cloud provider code from the main Kubernetes repository. For each in-tree cloud provider, we will report on efforts, accomplishments, and roadmap for getting "out-of-tree". We’ll also discuss the plans to handle the speed bumps that are left including removing disabling the cloud provider in upstream with a new feature gate and what to do about E2E testing.

A

All right, hello and welcome it's time for our Sig Club provider update uh if you are looking for Helm, that was last hour, so you can watch the recording. It was fantastic um but uh yeah, let's get into it, um shall we introduce ourselves.

B

uh Yeah, my name is Michael McEwen I'm, a software engineer at Red, Hat, uh mainly working on cloud infrastructure and auto scaling, related topics uh and I've been working in and around the kubernetes ecosystem for about six years now, and if you want to look me up on GitHub, that's that's my handle.

A

Awesome and I'm Bridget crumhout I'm a product manager at Microsoft, I focus on Upstream, open source and uh I've I. Think I've been around the kubernetes ecosystem for about six years. There's I think it. You know. Time has stopped passing since March, 2020, so I'm really not sure but um and I'm Bridget crimhoud on GitHub and all of the everything's. My parents made the exciting choice of giving me a Google, unique name in the 1970s good choice, thanks Mom and Dad, um okay, uh so spoiler alert.

A

You got to find out what this talk is going to be about and we're we're going to start I think we got to start with excitement and adventure. Oh.

B

A

I mean a Jedi craves, not these things but Sig cloud provider is all about them and uh then we'll do some cloud provider, specific updates and uh and then Community stuff and even more caps that you can dig into and get involved with all right. uh So, let's, let's start with the hey, you thought you could use cloud providers watch out. It's not going to be what you thought right. Give.

B

Some detail about that yeah. Definitely so anyone who has been watching you know the great migration of Entry to out of tree for our Cloud providers might be familiar with this cap uh 23.95, and this cap is the enhancement that proposes the removal of that code from the kubernetes.

B

You know main repo and then what's going to happen once we do that so currently we're in the phase four beta portion of that enhancement and that might sound a little overloaded, but we'll come back and talk about what that means, um and all of this right now is going to depend on the success of the integration testing, we're doing so as we're starting to remove the final pieces that will allow us to take these Cloud providers out of the entry.

B

um These tests are really important to what we're doing, and so what we need to do is kind of make sure all the tests keep working the way we expect them to as we move from one style of deployment to the other and.

A

I think that this is probably important for people to look at, because even if they think that they don't have to do anything, they almost certainly will have at least a few things that they need to do that. We can talk about yeah.

B

Definitely so like in in 1.26 what we're going to try to do if the tests pass is we're going to enable these feature. Gates, the disabled cloud provider feature gate and the disable kublet Cloud credential provider feature gate, so those will be turned on by default and those are the feature Gates that you would use today.

B

If you wanted to enable the out of tree Cloud controller managers as opposed to the entry, kubernetes controller managers and there's a couple there's a reference couple, references here at the bottom uh to the cap and then um to a related blog post that talks about the CSI migration, which is directly linked to this, because it has to kind of it has to predate, what's happening with the CCMS.

A

Yeah and and these slides are already on sketch, so you can take a look and off all these links. So you mentioned where we are in the removal, phases and I. Think it's a little bit deceptive and so I want to make sure everyone knows that this isn't a situation where oh we're almost done and everything's fine and it's basically nothing to worry about and we're just you know mopping up at this point, there's a little bit more going on that uh people should probably realize yeah.

B

No definitely definitely, and as you can see from this, this is enumerated in the cap, but there were four main phases we were moving through right and so, like we've kind of gone through. The process of you know, moving the cloud provider code from where it was in the kubernetes repo into the staging portion of the repo building, the CCM kind of library that allows us to reuse.

B

That code, then moving the code or really copying the code from the kubernetes repo into the specific cloud provider repos and now we're at this phase, four, which is disabling them right. But this is taking us a while to get through, and you know people have been asking for several releases now. When is this going to happen? When is this going to happen?

B

So 1.26 we're going to try to make it happen, but it might take us a few releases to complete it, because if we find errors in the integration testing, while we're doing this, we'll have to slow down our Cadence to fix those things. So I think this is something the Sig probably needs to talk about a little more. In terms of like letting the community know where we are in the process and how close we are to doing this because I've already been getting questions about like well, what? Where can I watch?

B

Where can I see when this is going to happen? So.

A

B

Is something we need to talk about.

A

And so I think the the part like you know how, for a long time, everyone said you had to care about IV IPv6 and you thought maybe not really, and that's it's been coming real soon now, for you know the the year of IPv6 and your kubernetes cluster was coming real soon now until it was actually here and it was actually happening and we're just letting you know that now this is actually happening. So, yes,.

B

We are actually trying it out now.

A

um Okay, so you mentioned testing and I think we should talk a little bit about that too, because that is key to how people are going to know. Are they going to be okay? Are you ready, yeah.

B

Absolutely and so most of the proud jobs that we run right now in the kubernetes CI use uh gcp the Google Cloud platform as the main line for running these things, and so as we're starting to transition. What we're going to do is we'll start by using that gcp CCM, which is the out of tree controller manager to start doing these tests and as as I mentioned, you know on the previous slide.

B

This is the highest priority for the Sig in 1.26. We were hoping, you know if the testing goes well, that we can actually deliver this and then you know the wider Community can start to use the auditory providers and get used to the style of deployment. That goes with that, and so the what we kind of like talk about here in the requirements before we could accept this is all of our pre-submit. Jobs will have to run with the CCM or no cloud provider.

B

So, as you know, kubernetes can be deployed in a manner where it doesn't have to be aware of the infrastructure provider beneath it. We're using gcp in our main testing so for the tests that require a cloud provider. Those will all have to run with the CCM and then most of the post. Submit jobs will also be running with the CCM or no cloud provider and I say most because there will be still some integration tests to make sure that the old kcms, the entry portions are still working.

B

um You know until those get removed and.

A

The and the note that you put at the end here I think, is really important. Important the we need to at this point start looking at individual out of tree providers reposed, for you know their specific tests because we're going to start no longer having uh one entry, you know to rule them all, but instead it's going to be variated okay. So what should people do like right now? If people think oh wow, this cup sounds like a big deal and maybe I need to actually take action. We have some actions for you, yeah.

B

Definitely so if you are currently running your kubernetes deployments in an infrastructure that has a cloud provider, you will probably want to understand how the cloud controller manager architecture works, and this first link.

B

It gives you an impression of how it fits into the kubernetes cluster and how it gets deployed in the kubernetes cluster as well, and this is, you will have to manage- deploying these controllers on your own, so, whereas before the kublet was handling this, for you now you're going to have to manage deploying these on your own, and so the next link the migration process.

B

Documentation would be really helpful to show you the process that you take to migrate from the entry to the out of tree, and this can be done in highly available clusters without disrupting work, and it will also show you what you need to do to deploy those on your own. So whether you choose to do it with a demon set or a deployment or whatever works for your organization, it will show you some of that and then my last call out here is start planning. Your migrations. Now.

A

Yeah, this is the sort of thing where I know. We all want to wait until the last possible minute, but we're giving you the warning that now is the minute to avoid having pain and sadness and I know. When also when people tell you, you should start doing this big amorphous thing, it's complicated and probably hard. The first question you have is: can you show me an example of where somebody has done something like this? Happily, our friends at Red Hat have yeah.

B

So, um as many of you may or may not know, openshift uh is a hybrid Cloud kubernetes, so it runs in many different places and at Red Hat we test it on a lot of different clouds. You know I think we test five or six different Cloud Platforms in all of our testing, and so we needed a way to centralize our transition or our migration from entry to out of tree, and we needed a way to share it with our users in a way that they could repeat these processes.

B

So we didn't have like a lengthy, uh you know, documentation they had to read or something. So. What made sense for us was to encode the knowledge about out how to how to migrate from entry to out of tree into an operator. You know we felt this fit really well with kind of the kubernetes ecosystem. It kind of fits into the ethos very well, and so what we did was we encoded all of our.

B

You know kind of logic around that into this operator and for us, this single operator runs on all of the clouds that we run on it can detect. What's Cloud it's running on and then do the migration and it works with highly available clusters. It will keep the cluster running while it's doing the migration, and you know this is something we test in our pre-submits and our post submits. You know daily basically, so it might seem like a heavyweight solution, depending on what you're trying to do.

B

But if you're operating in an environment where you know maybe you're serving multiple clouds or you have, you know lots of clusters going in at different versions. This might be the kind of thing that could help you to make this migration easier. Yeah.

A

And because this is open source, even if you're not going to use this specific one for your use cases, you can go examine it, learn from it right.

B

So yeah the link here, you know we call it. The cluster Cloud controller manager operator, like I, think our team is getting a reputation for making names that are like super long, and so we call it the 3cmo internally which I like because it sounds kind of like Star, Wars or something but yeah check it out. It's all there. It it works. You know you can copy it do whatever. Oh.

A

My gosh Mike I think 3cmo needs a cute mascot. You should get on that totally okay, so let's go move into cloud provider updates um and this is going to be we're. Gonna hear a little bit uh about what has been going on in each cloud provider, I think more or less since the last kubecon so about the last six months, but just highlighting a few of the things in the individual Cloud providers that are a little interesting or uh different or what you should know.

B

uh Yeah, so the AWS provider updates they've got a lot with us to share here so they've just had a recent release of their credential provider and they've migrated it from the entry. um You know, they've got a lot of bug, fixes they're working on some artifact builds. It looks like um they have an IM authenticator, which has new release versions as well and they're working on the next release version.

B

It looks like they're working on a bug patch that they're going to push out and then they've also got updates to their load balancer controller as well. um So if you're working on AWS, you know definitely check out the updates to their provider, all.

A

Right awesome thank you and for the Azure provider update I reached out to my colleagues who work on the Azure provider. They are not able to be here, but I wanted to put their GitHub icons and handles up there so that you could reach out to them individually.

A

If you wanted to because they're a great team and uh the recent achievements that they wanted us to highlight are we migrated all all of our test info jobs for our cloud provider from using AKs engine to using caps e we're using the cluster API provider for Azure to run all these tests highly recommended, and since you know, cluster API can be used in all of the clouds you've. Probably ever heard of and a few you haven't, you probably could check that out. The vmss flex will be supported in the upcoming release.

A

We've added Helm charts to deploy cloud provider, Azure improved error handling. You know various features just to kind of all of the small nuanced things to fix various bugs or add features that people need around like disk, IP Etc and then, of course, uh you know, CSI driver improvements and that kind of ties into the what's next. Of course, we are rolling the dual stack, ipv4 IPv6 networking that I mentioned earlier. That kubernetes has into collaborator Azure um some performance improvements.

A

You know Trace improvements, obviously like we all like to operate things when we can see what's happening so, um of course, Azure SDK. You know, updates Etc I did want to point out, because we talked about this a little bit. The removal of the Azure disk entry driver is coming in 1.26 and if you're thinking like oh wow, the CSI migrations can take a really long time.

A

It took us about four years, but after four years of work, we're actually done and we can remove that and then we're also gaing the Azure file, entry driver and so that's kind of, and then you know other again more improvements, Etc but I. Think that's a good example, and we will talk more about that too. Of the spoiler alert coming up a bunch of the stuff that you know you need to be considering, but yeah, that's the that's. The Azure update cool.

B

Sounds good and everyone go get a new get the new version of the Azure cloud provider right, so Google also has been working on. Some updates. um They've been continuing to kind of progress through CI and there's is kind of some interesting work. They've been doing behind the scenes pushing on like you know how to check different versions and everything, and you know, versions of the cloud controller versions of kubernetes. So that's kind of some interesting stuff happening internally, um their kubelet credential provider plug-in, which we'll talk a little bit more about that later.

B

They've created images for that, so they're all downloadable. Now um they've created this leader migration API to make their you know safe migrations for their KCM, which was the entry to the out of tree CCM in their ha clusters. And what they're going to be working on next is they're gonna. They would like to get a release policy out for the images they're creating so that you know people can know when to update and where to go to update. They would also like to improve their documentation and, of course, like testing testing, testing, I.

A

Feel like testing, if it's not on every slide, just imagine a testing testing testing on every slide. Yeah.

B

So we actually have a special guest here. um Our uh chair for the IBM cloud provider is going to come up and give the the update for IBM. So yeah come on up.

C

Thank you so much Mike and Bridget and hi everyone, so I'll. Just brief on the progress on IBM cloud provider. It's one of the sub project. There are a lot of hyperlinks here, so we already shared the deck with you. So you know all those links are reachable.

C

You can reach out to us on Sig cloud provider slack. We also have a provided IBM Cloud slack channel on the kubernetes lag, so you know you can reach out there we meet once a month, so it would allow you to join there, and then we have. Of course you know Google mailing list right.

C

So under Florida IBM Cloud we have four GitHub repos, the cloud provider, one, that's the CCM implementations. um It's always been out of tray, so we never! You know we didn't have to deal with the pain of moving from in treated out of tree because we started a little bit late in you know, 2018 time frame and that's the time you know this. We were started to going in this direction as as pleasure to Mike say it's been taking time. You know we're talking about out of tree for a good couple.

C

Three four years now: um cluster API, that's another GitHub repo. We have uh for IBM Cloud right. So there is a cluster API repo for life cycle management of cluster. That's a kubernetes repo itself right the project, and what we have is the extension of that for IBM Cloud uh to have the cluster management lifecycle management there. We also have two other repos uh one for managing like life cycle of IBM, VPC block data volumes and the second one is uh in a similar one, but for the Power vs.

C

um About you know, IBM cloud provider, repo itself. You know this is the cloud provider that that we use to run openshift on IBM cloud provider, IBM Cloud. We also use the same cloud provider to for our IBM cloud kubernetes service or what we call IKS.

C

um Some of the work on going. There is to make it more consumable, so refactoring is going on some Doc improvements are going on as well um and as of today, uh it's you know we're using it for openshift 4.11 to run on IBM Cloud. Using uh for running you know, kubernetes uh 1.25 is part of the IKS and I have provided the list of folks. They are working on on this different reports right. We have a uh one of the collate here, Brad toppel, so I'm glad he's here.

C

Others couldn't make it, but you know feel free to reach out to us and we'll be happy to. You know, connect with you and you know, help you as you need well. Thank you so much appreciate it.

B

Awesome thanks to Dev thank.

A

B

All right and lastly, we have the VMware provider update, so they've got a link to the vsphere cloud provider. um You can, you know Again download the sources from there check out their code, their latest developments, they so that for 1.25, they've done a bunch of bug. Fixes and they've also resolved some problems with their Helm charts. So this will make it a lot easier if you're getting into deploying these things they've.

B

Given you a little help here in terms of getting the image deployed in your cloud and getting it running properly, their credentials manager now accepts an alternate format secret for IPv6 and they won't delete a VM that is temporarily unavailable, I'm guessing this was a problem for some people and they now have an ed testing pipeline set up as well, so for them like many of these other providers what's coming. Next is more documentation and more testing.

A

Love to hear it, okay, all right! Next, we're we're going to kind of talk a little bit about Sig Community or to be more specific, we're going to talk about you because you're here, because you want to know where you fit into this and we've got some ideas.

B

So, first of all, I want to talk about like who we are I mean. You know who we are because we introduced ourselves already, but um in terms of who the the Sig chairs are and who the leadership of the cigar uh Nick Turner from AWS uh is one of our current co-chairs and Technical lead. uh You can reach them at.

B

You know: nck Turner, on GitHub and same thing on uh slack as well: Andrew cycam from Google he's also a co-chair and Technical lead um and he's Andrew psycham on GitHub and I think it might be slightly different on slack so find him on GitHub and then Walter Fender. Also from Google is our technical, lead and kind of Emeritus chair, uh Chef Taco on GitHub, and also on slack yeah.

A

And that's and we wanted to highlight them because uh while they are not able to be here at the conference today, they're very active and present in the community, so we want to make sure you know about them and know you can reach out to them or ask them questions. uh And then we also would be remiss if we didn't talk about the other sigs. We interact with the most.

B

Yeah totally so like we do have a lot of crossover and I think you know. Bridget and I were talking about this the other day and was like well what you know. What do you, what do you cross over with the most? What do I cross over with, and it was actually interesting that we both kind of like looking at different areas so, like I, think Sig, networking and Sig node? We tend to do a lot of crossovers there and we're always talking about you know things like you said: ipv4 and IP.

B

You know this is the century of IPv6 right. So.

A

B

Lot of talking about how do we, you know, find bugs there and everything because doing the CCM work we're doing a lot of these low-level like kind of networking things and we're coming up with some bugs and finding things and then Sig node as well, because there's a lot of interaction going on between how the cloud these new external Cloud providers, how they taint the node, are they untaint the nodes when they become ready and there's a little interaction going on there to set the note up?

B

And then you know, Sig auth is another big one, because there's always something going on with you know: I don't have my credentials. Yeah.

A

Yeah, exactly okay, um and so then we we did mention a big cap, that's going on at the beginning, but we have some other ones going on too. You want to tell us about a.

B

Mic yeah, so two of the other caps one of them is in progress right now and the other one is going into GA. So this.

A

Yeah so, like oh I, just wanted to say like let's repeat and like not forget that first one because I'm like.

D

We're jumping over it.

A

But we have more detail about the second two in the next couple: slides. But let's remind people: why do we care about removing entry.

B

Provider yeah totally totally I mean like yes, I did skip over the first one, pretty quick there, but yeah of course go back and check out 23.95.

B

It is worth repeating that we are going to try to remove the entry providers in 1.26. So if the tests pass, when you upgrade to 1.26 those feature, Gates will be enabled by default. So that means that the kublet will now be set to use external Cloud providers so be prepared for that.

A

So what do you expect? The experience of somebody who's just happily going along using a cloud provider of their choice um and completely ignores that and 1.26 comes and we change the defaults and then they're on 1.26.? What do you expect will happen right.

B

So today, when We Run, The kubelet, if we're using a cloud provider, let's say we're using Azure I will pass a command line flag to the kublet. You know minus minus cloud provider, equal Azure and that tells the cloud provider to you- know the kublet to run that cloud provider. Now, if I, don't change anything and then I upgrade to 1.26 and these feature gates are enabled, the Kublai is probably going to crash right off the bat because those Pathways aren't going to work anymore. So that might be the first thing you would see.

B

But if you had some other methodology, where you were, you know maybe using a different cloud provider or we're trying to use cloud provider none or something like that or no cloud provider. You, you may or may not see a problem here. It kind of depends what your interaction with the underlying infrastructure is. But the first thing I would look for, is you know, kubelet log errors for sure yeah.

A

Absolutely all right and then tell us what the other two are and then we'll go into detail.

B

Yeah so cap 26.99 is a cap. We're working on right now, we'd like to add web hook hosting to the CCM and I'll talk about that a little bit. Why we're doing that on the follow-up slides? But we need help reviewing that. So please come around and take a look at it and then cap 2133.

B

This is going to add some new functionality and it goes hand in hand with what's happening in 23.95 the removal process. It may or may not be important to you, depending on what platforms you're running on or what functionality you're. Adding we'll have another slide about that as well. All.

A

Right awesome, so let's talk about these web hooks.

B

Yeah so adding web hooks. um You know this is kind of a common pattern. If you're familiar with kubernetes you've probably come across web hooks, love them or hate them. You know they're here to stay I guess for now, but what we would like to do is add these into the CCMS and and the reason we want to do. This is because there is a volume label that PVS use this persistent volume label and it I think if I remember correctly, it has something to do with like the zoning of the storage or something.

B

But when these are created, you need to be able to have an emission web hook to kind of set things up and and make sure that everything is correct for the the kublet. That's coming up, that's going to be initialized. This also gives an opportunity to kind of run fewer processes in the control plane as well.

B

So you, you know you can kind of centralize some of these web hooks in a way where you could have a you know: a single web hook provider for the entire Cloud, so you can kind of reduce some of the workload that's going on there and there's a link to check it out to read more yeah.

A

And so that's a good one for people to get involved test it try it.

B

Yes, we need customers, reviewers yeah, for sure this one is still very much in progress right now and.

A

So, if you're looking for a place where you can start being a participant in City Club provider, this is a great one. Yeah.

B

Absolutely um and we got another one, so this other one kept 23 2133 about the kublet credential provider. Now this is going into GA in 1.26 and it's defining uh I, wouldn't say a new API, but but a way to to do a something that had previously been done in tree, so on AWS, Azure and gcp. Currently there is an entry credential provider that can acquire credentials to get container images from those clouds, the in much the same way that the CCMS are being moved out.

B

This mechanism is also being moved out and a an API is being created around the controllers. That can now do that work outside of the kubelet. So what this means is that it gives an opportunity for new Cloud providers, other Cloud providers- you know perhaps IBM or openstack or whatever just start, creating these credential controllers in the cases where you might need them and there's a little history here of it at the bottom.

A

Yeah awesome, okay, so back to Sig Community, we told us who you, who we are. We told us or sorry we told you some of the places you might want to get involved and uh here's where to get involved I mean probably the the first place to check out is the State Community repo.

B

Definitely yeah there's lots of good information there about meetings and like a lot of the information we have here is also duplicated there. So, if you check out one link like go check out that one.

A

um And we meet every other Wednesday, yep and uh I, probably I would say like the C Club provider. Slack channel is not too busy, but it's a great place to reach us at least I. Have the notifications turned on for that one, so I.

B

Tend to hang out there a lot so like you know, come and poke me or something yeah.

A

And breadcrumb- and this one this one which is say you you actually think wow cloud provider I, want to get a little more involved with this wear something with a little more substance that I can dig in and really start learning and understanding. Yeah.

B

And so this last link to the cloud provider repo. um This is like a library or framework or whatever you want to call it that we that all the new Cloud providers are built off of- and this was this was created by distilling the logic that existed in tree and pulling it out.

B

And then what that allowed us to do was take all the entry providers, use this new external library and move them all out built on top of that, and so I think it's kind of interesting just to learn about if you're really into like kind of getting into the tech. It's a great thing to become familiar about, become curious about it, and then you know maybe build your own cloud provider or help out or whatever you know.

A

Yeah, absolutely okay! So um that's uh pretty much what we got! um Thank you much for uh joining us yeah.

B

Like scan this link like give a happy face or a sad face, if you didn't like it or what ever and I'll be at the Sig, meet and greet this Friday uh so which I guess is probably just right around the corner from here. The 411 I guess.

A

I think it's right over here.

B

Yeah so come say hi if you've got any questions or want to know how to get involved or whatever, like yeah, we'll be there for a couple hours hanging out all right. Thank you.

A

And we have a few minutes for questions if people have any questions.

B

I'll take them again all right jack. This better be a good one. Man.

D

Hi thanks everybody. um So when you are adopting the new out of tree cloud provider, is that just a kind of hygiene exercise for your for your cubelet or are there new features and new bug fixes in out of tree compared to entry.

A

So that that is a nuanced and interesting question, because, as Jack is correctly pointing out, once we've uh digressed from the entry single code base to rule them all, yeah I'd pretty much guarantee that every out of tree provider is going to have different, exciting new bugs.

B

No yeah, absolutely right, plus one to everything, Bridget said and and I think. The other thing that's interesting to note about that, too, is that over the last few Cycles we have been not accepting new patches to the entry providers, so the out of tree providers, depending on which accessories.

A

B

A

Been chasing a lot of like, and now we should backport this cherry pick of this bug fix. Let's.

E

A

Reviewers, who can get it into this particular release.

B

Yeah so, like so I think looking forward the the auto tree, Cloud providers are probably getting features that the entry Cloud providers have not been having, because you know, like Bridget's, saying we're only taking security back ports into the into the entry so and in the future. That's where they're all going to go so yeah.

E

Thanks for the great talk, um so I was talking about maybe a follow-up to the previous question. um You know one of the things that happens whenever you decouple you know code is now you have a synchronization issue. um Do you have like I? Don't know testing policies to in place to like make sure that these new auto tree things stay synced with the latest Cube and everything.

A

E

A

We started I think we started didn't you start with a discussion of testing like this is thank you you're, taking us full circle to uh why this tells us exactly why Mike said testing is going to be Paramount yeah.

B

Totally and I think there's actually an interesting kind of detail here and it's you know we don't have a slide for it, because this is more kind of like a for lack of a better word. Like a Skunk Works project.

B

That's going on inside Sig cloud provider, we've talked about it for several Cycles, but we've never actually kind of achieved it and we were working on something called least known good, and this was going to be a testing Matrix or it is a testing Matrix we're trying to figure out how to make it work where we can test versions of kubernetes against versions of the cloud providers, and so we can we tricks of different ways to test it, and so the idea would be that we would.

B

We would do the least known good testing, and from that we would. We would have a list of where you know problem areas are and whatnot. So we have some ideas about it. I think, aside from the integration testing, we have going right now. You know that'll, be our that'll, be kind of like our red line for the moment and then in the future. I think we'd like to get into this least known good.

B

It would probably be most useful as we're going through this, like transition phase from the entry to the outer tree, but it could be useful in the future as well just to kind of create this Matrix of options so yeah. Thank you good question.

A

And that's probably a good place to um oh, we have what we have one more question from over here and then we'll uh I suspect that this is uh gonna help us segue to the.

F

Where to go, maybe not um thank you. uh This is really interesting. I was just gonna. Ask now that all the club writers are kind of doing their own thing in their own repo.

F

um Do you see an opportunity for them to collaborate together and stay in sync to ensure, like consistency- and you know, user experience that translates from one Club provider to another.

A

And I think that we can take example from other cigs like a Sig cluster life cycle um like I know it. The Cappy umbrella project certainly informs a capsi and all of the other cluster API sub projects. I, don't think we have that kind of structure yet, but maybe we need to be thinking about growing. Something like that. What do you think Mike yeah.

B

No I mean I think it's a good idea like you're. Absolutely right like right now. You know so I go to a lot of cluster API stuff too. You know shout out to my friends here: um I think the cluster API Community has like a much stronger idea of like how to keep Cadence between the core of the project and the and the kind of the individual providers.

B

You know we do have meetings and- and you know we attract people to come and ask questions and and talk, but we haven't created the same kind of mechanisms for doing that. Collaboration, I, think the cloud provider is the main point of collaboration. I didn't realize that went off um and so probably like enhancements would come there, but then each individual cloud provider could start to vary and it might be tough for us to kind of organize that so.

A

B

Think it's a cool idea, something to talk about for sure.

A

I, like the idea and I think that maybe that just takes us back to our message of testing we're going to probably need to create a reasonable set of a test Suite that kind of helps, ensure minimum consistency is met. Whatever we're determining the minimum. Consistency needs to be like oh gosh, this cloud provider doesn't pass any of the tests that the other Cloud providers pass. Maybe we need to get some more viewers in there. Maybe we need to get some more maintainers in there cut.

B

Them out yeah for sure and it'd be you know, I think our meetings are pretty lightly attended. Now too, so it'd be great just to see more involvement from the individual. uh You know Cloud operators and whatnot, and just you know we have a group about I, don't know six to ten of us that meet pretty regularly, but it's always great to see a lot of activity there. So I'm here, for it.

A

All right so on that note, I, think we're about out of time, and so I just want to. Thank you all and tell you. Yes, please take a look at say cloud provider. If that's interesting to you, because it's definitely a place where you can jump in and make a pretty big difference pretty quickly. Absolutely.

B