Kubernetes Cloud Provider Special Interest Group, 21 Sep 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cloud Provider migration/extraction subproject 2023-09-21

Description

Meeting agenda: https://docs.google.com/document/d/1KLsGGzNXQbsPeELCeF_q-f0h0CEGSe20xiwvcR2NlYM/edit#bookmark=id.z8ql2m3k2ai

A

uh Welcome everyone today is Thursday the 21st of September 2023, and this is the Sig cloud provider migration extraction sub meeting. uh This is a sub-project of kubernetes sigs and as such, we follow their meeting guidance, which basically means uh please treat everyone. Kindly and I won't ask you to raise your hand Andrew, since it's just you and me um yeah, so we're talking about uh cap 2395 and the update for beta status, which I think is I, think is achievable.

A

um I've got the new template in place and I'm, starting to kind of move around some of the information to fit that template. The things I'm concerned about is um well one October 6th is our deadline to merge the enhancement, so we've got just a little over two weeks to get this done and then um I think the parts that probably still are going to need, like the most work, are like.

A

There's these sections by unit tests and integration tests and E to e-tests, and they kind of ask for some links to coverage and whatnot I, know like in the ede test, we're pretty good I, don't know if there was any unit tests or integration specific tests um that we should be like linking here.

B

Or whatever um I mean for unit tests, you can basically link like any unit test in the cloud provider, um repo for the controllers and and what not. um Integration is harder, because yeah.

C

We don't have a lot of.

B

Integration tests I, don't think so, but.

A

That's not necessarily a problem. um Can I just ask you a question about the unit test.

C

A

Are you talking about from the sample cloud provider repo or from like cloud provider AWS or something.

B

A

B

um Let me find.

B

Come and see here, uh yeah like one example, could be like this.

B

Because I don't think like, like.

C

B

Coverage uh like no one expects the unit test to cover like the actual integration of external providers, but we should test that the controllers that we use for the external providers are like well tested so.

C

As long as we are like.

B

Unit testing, like individual controllers, that are part of the overall architecture, I think that's fine. Okay, that's just there's a lot more tests in there. So, whatever tests you could find I think just like those.

A

Okay um and then yeah for.

B

Integration tests like uh I, would just say for the integration test like just we. We skip integration tests because we found more value and end-to-end tests and, like we have plenty of end-to-end tests, so we didn't feel like integration tests were needed, or maybe like a more better answer, which is so I I feel like the the real reason why we don't have integration tests because, like we're.

A

Talking about testing against, like.

B

Real infrastructure providers and like it.

C

Sounds like such.

B

A pain to like try to mock provider and integration test um like we could we.

C

Could we could create.

B

A fake provider that runs integration tests but like how valuable was that really so just.

C

Like yeah, like.

B

C

Do everything end-to-end with the.

B

Real top provider- and that was kind of how we've been doing it and yeah.

A

Yeah that makes sense to me: I mean I kind of I was kind of figuring. These would be the answers to some of these, but I just didn't know for sure and uh welcome Kirsten um yeah, the ede test. I think like do we do we actually want to link to all the ede tests that we feel exercise the the controllers here or.

B

um So I mean I think the first thing we want to say for the EDB test is to say um we.

C

Actually have all.

B

The like, by default and proud of all the edu tests, are now testing with the external provider. Well,.

C

That should be the first thing we.

B

Say because that already is um stating that we have quite a quite a lot of coverage already yeah, um but then in addition, I I would just link out like um we.

C

B

uh One we we have like two jobs that were specifically testing with Club provider disabled before we did the switch, so I would link to that, because that that job proves that we've been running like we've, been testing. This configuration for a while now and we just took the switch recently.

B

C

Me I'll dig it up for you in a.

B

Second um and.

C

Then the other test, I.

B

Will link is the recent tests, the recent crowd job I added to pull in the latest kubernetes libraries and cloud provider, gcp and I know dims also said: AWS added a periodic job to do the same thing so I would link those two, because those are more like supplementary test coverage that we're going to use as additional.

C

B

Let me let me pull up the jumps. um Okay,.

A

And welcome Caleb we're reviewing the updates to cap 23.95 right now. uh Caleb.

A

um So, just to kind of like dovetail on what you were saying there Andrew um there was nothing.

A

We had nothing in the test plan section before I think that might have been like a new section that was added or something and so I added some language here to talk about how we're going to leverage the existing tests that we already have um and that you know this change to the new Cloud controller managers is not changing like anything behavioral in the way users use kubernetes, it's it's just changing like a configuration, so those tests should still be good.

A

You know once we switch to the external stuff, so I tried to put some language in here about that. That.

B

A

um And then I also added a note about, uh because there was a I. There was something here: I added a note that, like we're working to expand the tests but I didn't think those tests would be like a prerequisite for this work. That needs to be done so just kind of signaling to people that we are expanding the test. But you know we're not planning new tests as part of this cup.

A

B

A

Think it's worth mentioning that or should I not even mention it.

B

A

We've added a lot.

B

Of tests over the over the years related to this so I wouldn't say that we're not adding new tests um but I think it's worth saying that our focus is going to be preserving like all existing behaviors um before the migration, so just ensuring that, like no con, we're not breaking compatibility or we're not um changing any behaviors. That would be unexpected.

B

C

B

I wouldn't say that there's a new test, but I would just put the focus on like we're making sure all the existing tasks are passing and yeah good shape before we do. The switch.

A

Yeah so, like I just said, no new tests were planned as part of this cap, but we're continuing to work. You know I guess maybe I should change this up a little bit. Yeah.

B

I, wouldn't even mention that I would just say, um but prerequisite is that we are ensuring all existing tests pass.

B

Because if you, if you say like oh we're, not planning anything we'll like it's well, first of all, it's not true like we have got a test, but also I don't want to like people questioning. Oh, we have no test coverage, it's like no. We we have a lot, but just not.

B

A

Okay, yeah I'll, take I'll, take out that part about no new test planning and just I'll make sure to to reinforce that. We're like the prerequisite here is and share ensuring that all the existing tests are passing yeah. What I wanted to kind of what I wanted to note was like I guess, maybe you know, there's a difference between the tests.

A

We have that are non-cloud provider specific like some of the service tests or whatnot, and then some of the cloud provider specific tests, like you know, like the node lifecycle tests on you, know, there's like going to be a difference in how those get run. You know yeah.

B

It's also worth mentioning um that individual providers have their own Suite of tests that all run using the external configuration. So I would also mention that, okay to say, like.

A

B

Core kubernetes we've done this like effort to ensure all the existing tests and conformance or can all passed using this external configuration. But we also like delegate to individual providers to make sure that they're testing this configuration extensively and they have their own tests to test their own implementation.

B

C

B

If we need references like all the repos have plenty of test references, we can just point to all of them.

B

I'm trying to find so Daniel's mentioned AWS having new periodic jobs, but I trying to find where the task grid link is foreign.

B

Aws ewe, Cube plus two.

B

I'll just put in the chat, let me first, but so this test from gcp. This is the this is the job that pulls in the latest kubernetes source and compiles the cloud controller manager and runs edu test.

B

C

That's additional.

B

Coverage that basically ensures latest kubernetes is not breaking like a downstream implementation and then I don't have a I, don't have like links to catch results, but bims mentioned.

B

C

Is what dims mentioned in the.

B

B

So Cloud CI collaboration.

B

Is supposed is supposedly pulling weight as KKK, but I don't see any references to this.

C

B

Look up the test grid, one second, um if.

C

I, look up that job I think I can find another.

B

B

Okay, here it is.

B

A

Okay, cool yeah.

A

um Okay, cool that, like definitely helps out in the Test Section I. Think the other thing that I'm a little concerned about is this graduation criteria as well, because I'm just not sure like I'm, not sure what what we actually did in the alpha part that we could talk about and then kind of what would I mean I guess I could I could figure out what we're doing in the beta part now, um but I'm really not sure like how this is supposed to be filled out. I guess.

B

A

B

For Alpha I would basically just say um sorry I, think no I, don't um so for Alpha it's. Basically, uh when we added the picture gate, we were basically in a place where we had very stable implementations um of, like all the external providers, so I would just say, um like stable working uh other tree implementations for the existing entry providers.

B

With you know extensive testing and whatnot, um and then the beta criteria is really just um like beta is, is the biggest Gap. The beta is the biggest jump because we're default stabling, the cloud providers um so for beta I would say um so. The criteria is basically going to be most if not all CI and kubernetes is using external provider which, which is which is complete, um and then in addition to that, we have um multiple providers testing the latest kubernetes libraries. So we have that as well. Well,.

A

So could you say the first part of that again.

B

We have multiple providers that are pulling in the latest, um kubernetes libraries from Master and and doing like edu tests on those um sorry, sorry, the the reason why that's relevant is because, for the previous point, um we basically switched all the careers yet to use external provider, but we're using we're pinning it on some like known, good version right and so the last point is relevant because, uh because we're no longer because we're pinning it on a known, good version, we need to also ensure there's coverage or whatever changes that are pushed into master in the kubernetes.

C

um So I don't know how you want to work.

B

That but yeah that's kind of the main idea.

B

B

And then I would say.

A

I guess does that make sense if I say like since we're pinning to known versions, we also need to ensure that we're not breaking against the tip of.

B

This okay, that sounds good.

B

C

B

Would also say like uh extensive documentation and communication about or something along those lines like we're going to break a lot of users. So, let's make sure we have the right documentation. Please.

A

Okay and I guess: I: guess we don't need to fill out the ga stuff yet or do we need to fill out GA here as well.

B

uh We could I mean we don't have to, but I would I would fill it up, because the g81 is really easy, like I think GA is basically like. Oh, we, we went beta for like two releases, and things are okay, so let's just put the switch to J um so.

C

I, just I don't know I just really like.

B

One or two releases of soak time in beta something like that.

A

B

A

Like do we want to see like I think the is there any code left entry or should we say like removal of all like.

B

Yeah I would uh yeah that's it's.

C

B

We want to remove the actual source code on the same release. We go ga or if we want to go ga like way to release and then like ask all the code, uh because for whatever reason, if we wanted to backtrack and uh flip the switch back to Beta and we've already removed the source code, like that'd, be a huge pain. So.

C

B

B

Yeah I don't know.

B

B

uh Under the deprecation board,.

A

Well, I, guess: that's deprecation of this feature.

A

I don't know, I was trying to understand how that they have like in the you know in the graduation criteria. The last point is the deprecation part and I. Don't know if that's talking.

B

About that oh I see yeah.

A

B

A

I guess yeah: you could put it in.

B

There, like I, don't know I, think I think uh that might be referred to, because future Gates now can be marked as um there's a new phase for future Gates. There's like Alpha, Beta, J and deprecation. So.

C

B

Applies if you're, using the deprecation phase, which.

A

Right, okay, I didn't know if we needed to mention the deprecation of like the flags that'll, be on like the kubelet and stuff like that, and those kind of things.

B

Yeah yeah, we probably should we probably should I.

C

B

It's implied with the state of the future date that it's deactivated.

C

B

A

I guess: are there any other flags we're going to remove aside from cloud provider.

B

Would we remove yeah we're not going to remove we're now removing the cloud provider.

A

B

We're just removing the option to set it to whatever the entry options are.

A

um Okay, then I realize we're cutting through time. Pretty quick here here. I, don't think there was anything else on the agenda for today, but let me just double check.

A

uh Yeah, okay, just this okay, um I, think I got a couple more questions here, uh so upgrade downgrade strategy. um I've been trying to write this part, mostly today and I'm, trying to leverage the documentation that we already have created. Upstream um like the cloud controller manager, Administration, docs and the you know, migrating replicated control plan I've also tried to link to each of the providers that formerly existed as entry to their specific out of tree CCM doc.

A

I could not find one for GC, though I don't know if something like that exists, but I didn't see it on the cloud provider. Repo.

B

Yeah I probably should.

A

And then I'm, bringing together like uh I'm, bringing together the the guidance that um dims has been putting together on that issue in the KK repo I'm, trying to figure out how best to format it here, but I'll also include a link so that I guess users could check that um so that that's kind of what I'm working through now and I I think it I think I'm. Okay, with this stuff, like I'll, we'll just kind of discuss it on review or whatever I, don't think, there's much to go into there.

A

um But this, like version SKU strategy like do we need do we need to handle this or.

B

B

I think the Virgin skew is handled by the.

B

The CCM migration stuff.

A

C

A

Expect version SKU to be depend like each provider's CCM is where the versions queue would take place right, like they're, going to be responsible for making sure that their cloud provider is up to date and in other respects we would just follow the normal. The.

B

A

B

Yeah so I would just add a note that I think we have this documented somewhere else, but the version SKU for cloud control manager is effectively the same version: SKU policy for cube controller manager. So um and then you can also mention that each provider publishes their own like supported version and their skus as far as I can tell- maybe maybe not for gcp but I, know some other products. Do that so um yeah.

A

Okay, so I think that really answers a lot of my questions and thank you uh Andrew. Oh, you know what do we want to do? Do you want me to try and put some stuff in here for drawbacks about this.

B

B

Yeah, you could I mean I, think the biggest drawback is the amount of time and complexity. The whole effort was but like that's already something, we've paid the cost for I'm, not sure, like what other drawbacks.

C

B

I guess, like the other drawback, is toil and um potential um operational load on users, because they have to do for those who have to do the migration themselves and not using like managed Solutions. So.

A

I I guess that's a legitimate.

B

A

B

A

I didn't touch any of the alternative stuff.

A

Just trying to I'm just trying to look for where I, where I've added content. Here, because there were some sections we needed to kind of fill out a little more.

A

Yeah, okay, so the other kind of big change that I made from the way we had it before was um in the new format in the new template format. There's like a proposal section and then a design details, section and so I I took what we used to have in what we called the approach, which was like a summary of all the changes. I put those in the proposal, as well as the risks and mitigations, and then I broke out the individual phase details to the actual design details, part of it yeah that.

B

A

Good, okay, um yeah so I think other than that I think that's! Those are most of the changes I've made so far and then just trying to get the release sign off check leaves filled out too I'm. I figured that'll take some more work from the community, but I.

C

Just tried to fill out the ones I know.

A

We've done already yep sounds good. Okay,.

A

All right, well, I, guess like that was that was really all I had um and thank you Andrew for all the help there like I, think, hopefully um maybe by tomorrow or Monday I'll, have the latest version of it. We can start reviewing it then and kind of get to the next part um sounds.

B

A

um Are there any other topics that people would like to talk about in the last couple minutes here.

B

Oh I guess one thing uh to worth calling now: the um external project for the um persistent volume admission, labeler uh I've, been on call for uh the past a week or so, and so I've been I, haven't, been able to look at that, but I'm really hoping that um before even the kept deadline the prototypes is like or the the the project is a viable like replacement, and we can do more testing with it and I guess.

B

The the way to know we're done with that is we'll have to integrate it into um kubernetes ACI. So we'll have it like installed by default and.

A

B

A

And then you want to bring like a mention of it into this cap. Basically,.

B

Yeah likely yeah.

A

B

A

Sounds good, although you're going to be like you're going to be burning the midnight oil to get that integrated like up and running in NCI in the next two weeks.

B

Yeah we'll see uh yeah I need to free up some time with some internal stuff to figure out when to do that. But yeah I'll try to find the pamphlet.

A

I hear that okay, okay, cool um yeah, nothing else, Kirsty and Caleb. Anything from YouTube.

C

uh Not just specifically same thing on the same topic like I was mentioning like in the thick liver. In fact, I started working on the uh Facebook extension today, so yeah I. Imagine yes, I was testing. It looks good, so yeah, hopefully I'll try to get a PR by before the next day, uh like next Wednesday submitted, try to.

A

See yeah sounds good Caleb. Anything you want to add.

B

C

No nothing from my end. Okay,.

A

Great all right! Well, uh thanks for coming out everybody- and hopefully we'll have this cap into at least like a reviewable state by next week, and we can inch our way towards getting this into beta State thanks again for all the help. Andrew all.

C

Right cool take care, everybody.