Kubernetes Cloud Provider Special Interest Group, 29 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cloud Provider 2021-09-29

Description

Discuss Webhook support in CCM KEP.
Kubecon planning
Mixed protocol in service support for cloud providers.

A

Welcome to the september 29 2021 sig cloud provider meeting, uh this meeting is being held by sick cloud provider kubernetes and the cncf, and we adhere to all of the standard practices of the above. uh A lot of that is going to boil down to please be polite, can consider it inclusive of all of your fellow contributors.

A

uh With that, I'm going to go ahead and start.

A

uh Share my screen.

A

I guess that one yep all right, uh starting with the sub project updates or the up project updates. Apparently I don't think I see anyone from alibaba or baidu here. uh Amazon do we have any updates.

A

B

I have anything.

A

All right, uh I did notice that we're uh changing the owners slightly for uh the aws system.

B

Oh yeah, that was one of the owner's files that I missed. I think it was in. um I guess the case I o repo, um so there was some inactive contributors, um so we removed those- and I put myself and matthew wong, um and I think I need a lgtm from you walter on that one uh you've gotten it already.

A

Oh perfect, no worries uh anything else.

B

No, I don't think so unless ki sure has anything I'm good. I don't have any updates for today.

A

Well, um azure.

C

Just I dropped a link into the one thing that I came to chat about, which was um this is actually a good idea. A good example of this is already fixed out of tree and reporting the fix back to entry um yeah, just cherry picking in the fix need to get that approved emerged. I think we were looking for andrew, but I suppose anyone could take a look and see if it works for them to bring that to people's attention. Yeah.

A

No worries, please send it to me uh it's actually. There are very few people who can actually approve this.

A

I think there are a couple of the top level approvers like tim, hawkin and dims, and then there's andrew and I and I think that's about the extent of it so.

C

A

Shall I send.

C

It to you on slack or where would you like it.

A

Slack is you, can you can either uh sign it to me in the bug itself? uh Chef taco or you can ping me on slack either one works. Thank you. Yup no worries, uh gcp.

D

I'm trying to think what to add here um we did bump to what is it 121 of open source kubernetes, and there are some networking changes going in.

A

Cool sounds good. I don't see anyone from huawei ibm.

A

I thought I saw el miko.

E

Well, yeah, but I'm not doing the ibm stuff rich, I mean richard would be the coverage there. So, okay.

A

No worries: uh do we have anyone for openstack, I think so.

F

Vsphere uh yeah. Sorry, I'm not prepared to address that myself, because I thought andrew would be here and uh not going to tell you more than I know so. I'm going to have to not provide an update on that.

A

No worries um so extraction migration. uh I think we have and we're starting to build consensus around the lkg uh did either joe or kermit have anything they wanted to add.

D

um The design is available, I think, there's a link further down uh so yeah anybody that doesn't know what alaska that's. No good testing is or does know, but wanted to provide us input or thinks they might want to consume. It recommend reading the doc add comments there. I think that's the stage we're at just looking for feedback.

A

Yeah, uh the other thing I will make a general comment on. I should probably sorry I'm talking and not typing, I'm not good at doing both at once. uh We have one kept for this coming release and very little time left we're hoping to start uh adding web hook, support to the ccm that's what the predominantly the cap is about.

A

A lot of that is specifically for the goal of allowing storage to provide a web hook for uh legacy.

A

uh Persistent volume claims support it, so the kep is, is approved for alpha and uh there's a fair amount of detail on the work that is available. uh So if anyone is interested in doing in getting started, please reach out to me, I am happy to help. uh I'm also happy to I'm also hoping to do some of that work, but there's quite a bit of work to do.

A

I think we can break it into chunks and I think it is a great project getting started if anyone is interested, uh and obviously that is one of the things that we need to do to finish up the extraction, migration work and thank you, michael for the typing, uh also just fyi. It's uh taco with a k.

A

Thanks uh sorry for that uh agenda steve, I believe you're up.

F

uh Okay, so we're planning a maintainer track session at kubecon coming up in two weeks, uh walter nicholas and I are planning on being there live, but we have a good sized portion of that session set aside for what we're calling cloud provider lightning talks, so these would be updates from the individual specific cloud providers, with the expectation that this would be a three-minute summary of where you've been in the past in months.

F

What state you're at now and what the road map looks like going forward, particularly uh with regard to migration out of tree.

F

uh What we'd like to do is get that lightning talk uh submitted in the form of an mp4 video uh high def 16x9 would be great if possible, and then I will, assuming I get them at least a few days before I should be able to edit them together, so that we have minimal disruption just hit play one time and let them roll rather than digging up individual files from the stage when we're up there live. I think it will be better if we get these in advance.

F

The other thing I'd like to get associated with that. If in your video, you show slides, um give me the slides, because the conference expects that a pdf of the slide deck gets published and I'll cut and paste whatever slides you used into the main deck for the session, so that an attendee who downloads it later we'll have. You know the appropriate experience of being able to look up anything they saw for reference later.

F

G

F

About it, I'd like to have them I'll, say by monday of next week.

A

uh Yeah, I let me just plus one everything steve said a couple of minor points, even if you're attending I strongly recommend having them before and getting it to steve the traditionally.

A

The uh broadband at the conference is terrible, and you know we don't having all of this pre-built and ready to play is going to make your cloud provider look better because you're not going to be sitting there hanging on a broadband and making it look like your cloud provider is slow, so strongly recommend getting steve those those pieces and and not trying to do anything with your actual cloud provider live at the conference uh where possible. It would also be nice.

A

I I think a lot of people are really interested in those timelines, so the you know it isn't important to touch where you've been in the current state now. But I think you know that road map is the thing most. People are most interested in.

F

Yeah, the other thing is now that walter brings it up. I think, even if you are intending and gonna be one of the moderators like me uh for the vmware cloud provider, that's going to be recorded because with covid going on, you never know what might happen.

F

Maybe you're thinking that you'll attend up until the day before and something goes wrong, so not bad to have an insurance policy.

A

Absolutely peter.

H

Hi, my name is peter: I'm uh maintainer of the k, ops project, uh so we have a lot of end-to-end testing in place uh and we're a lot of.

G

H

uh Testing, our external cloud provider, uh external controller manager, components and we've discovered an issue recently where the aws provider added a new field related to ipv6 in its cloud config and what's happening, is uh we when we use the external uh controller?

H

uh The api server still relies on the cloud config for the uh pv admission plug-in, and I believe this directly relates to the cap that was mentioned earlier, and so what happens? Is the entry code doesn't recognize the new field and it uh fails the admission entirely, and so I'm wondering uh how we should handle this.

H

uh I had a few proposals and I guess it might depend on um the timeline of having the web hook implemented, um but one one of nick's suggestions was to.

H

Have separate cloud config files for entry versus out of tree? uh I also suggested you know, depending on the timeline again whether it makes sense to keep the schemas relatively aligned for entry until we can drop the entry code. But I'm curious what other people think about this.

A

Hey nick, I think yeah, I'm going to put you on the spot. Sorry, okay,.

B

Yeah no worries um well, like I suggested here, you know in the in the cops test. What we could do for now is just have a separate cloud. Config for you know, have one for api server and other entry components and then then use the old schema and then for the external ccm we can.

B

We can have us, you know its own, its own cloud config and then, as soon as we have the the um admission controller broken out um and running as a web hook as part of the ccm, then we don't need to pass cloud config to the api server anymore um yeah, and I agree that there's the other option, which is to keep the entry um cloud config in sync, but I think we that sort of goes against the current policy of not merging features entry, and you know these will be.

B

You know, changes to the cloud config would probably be considered features. So as long as there is a workaround, um I have a feeling that walter and andrew and others will prefer it uh prefer that we use that. But I don't know what do you think walter? Do you agree with that assessment.

A

I it sounds right, I have to admit I'm not as familiar with the aws and exactly what's going on here. So I'm a little it's one of the reasons.

B

So let me basically in the test, there's and and peter please correct me if I'm wrong, there's one cloud config file right and it's shared by all of the components, and we have a new feature in the external ccm which uses a new field in the cloud config, and so that cloud config is also passed to the api server which, uh because of the persistent volume admission controller, is, is also reading the cloud config.

B

uh You know it shouldn't be, but it is because of the the pv library or whatever it is yeah yeah. um So you know the workaround is really to to use. So thank.

A

You to really the kcm, then am I missing something because it's uh I'm sorry they're, just I'm trying to put all the pieces together in my head. I would imagine and correct me if I'm wrong the cube api server is ignoring the cloud config file, so it's probably only the case cm uh that is paying any attention to the cloud config file and it's doing it specifically for the persistent volume controller.

H

uh No api server does recognize and use the cloud config file because there's an admission plugin purely.

G

H

G

All right got it.

A

All right now, sorry, now, I'm I'm a little optimist. Speed.

A

Yeah, I think then I have to agree with your assessment. We can. We can double check with andrew he's, usually the most hard line on this and you've got to get both of us to agree, and I don't think I andrew well, so I think you're right if there is a workaround um uh yeah okay, I mean this is obviously also going to block the um the credential provider featuregate, so uh yeah. I think we just, I think, you're absolutely right on your initial assessment peter.

A

This is just further increasing the importance of getting uh the web hook. Support into the ccn.

A

I mean, arguably and again, I'm gonna throw throw nick under the bus. Sorry, nick not your day, uh you at one point nick, I thought were considering uh the idea of actually fixing this with a con with your own controller rather than a web hook.

B

A

B

I I my memory is that um uh that was proposed by somebody else at aws and I don't I don't know fully the reasoning. So my memory.

A

Of the reasoning was that if we make it a web hook, we always have the standard web hook problem that if the web hook isn't functioning that you know you start having problems where either the mutation uh isn't applied or where you end up failing the right, and so whoever I thought was that amazon was preferring a controller route because it ends up. I mean it's, it's async, so you have some async problems, but you you, don't you don't end up? You know either failing the right or failing to do the mutation.

B

Yeah I mean the the web hook. Issues are obviously annoying. um I've had plenty of those. uh You know um they cause on-call pain operations, pain for us, but um if there's like a race condition which I thought there was um and that's the whole reason that I mean the controller approach like I, you know- I don't know the details of this, but my understanding is the controller approach was initially the uh proposed replacement and then it was rejected because of some race condition.

A

Yeah I I know that uh on both side, ali and michelle al, uh both googlers on this but are working principally on the storage side- were not happy with it as as the gcp solution, and so they were looking for. A web hook um got it, but it is, I mean at some level we'll we will.

A

You know for us for that to happen. We push through the cap. uh That support should be available to everyone, but doesn't mean that everyone has to go with that solution. uh But if you prefer, then I think that just increases the import of getting this cap into alpha.

B

Yeah, well I mean maybe I mean I'm assuming that aws does not differ from uh gcp. um So if, if we have the same race condition, then we would need to do the same thing, but maybe maybe I can try to figure out who uh was was going for the the controller approach and see if they know more than I do.

A

Sounds good all right! um Thank you, nick uh all, right, if we're done there, bridget.

C

Yeah I added another issue because I thought with this community here. This would be a good crowd to talk to about um there's. There was some discussion both in the issue and at the club provider. Azure meeting andrew brought up this idea of what, if we move this to beta and 1.23, because you know we haven't hit the exception deadline, yet um the thing about moving it to beta is, of course, as we all know, beta features around by default and then there's some back and forth there about. Well what what do we want?

C

The experience for users to be, but at the same time this has been. You know, january 2020. This has been going on for a while. So are we generally in the community in favor of moving this to beta or yeah? Basically, I was just seeing what people thought um I'm fine with filing the paperwork. I just don't want to cause unpleasant surprises for other clouds.

C

It looks like tim hawkin is not against it and I'm not sure if our aws friends have weighed in um yeah, I feel like I'm, I'm the caller first, you know first time caller I'll, take my answer offline, I'm not asking for an answer right. The second I'm just kind of asking people to look at and consider this maybe put their two cents two cents in on the issue.

C

Yeah, I'm sure walter's probably seen this one before so he has thoughts.

A

I haven't seen this one before andrew has been handling it.

A

So my first thought is: I I think, you're approaching this the right way.

A

There's something for me: um uh yeah go ahead.

C

A

Just saying yeah, it looks.

F

C

Are some comments? Yep bowie's commented, um I think tim's definitely commented. There's been some aws comments but yeah. Basically, I just wanted to bring it up in the meeting so that people had had a chance to not be surprised by it. If we do end up doing anything.

A

D

It looks like you already have a bunch of the right people on there.

A

So I think from the cloud provider's perspective, um I would like to get each of the major cloud providers to say they don't have a problem with it. uh If you could, if you've gotten, bowie and tim to say yes, then gcp is obviously okay with it.

A

uh If you know uh from there, I'm assuming that, since azure is pushing for this- that your azure is okay with it uh from there, it would be nice to if we could get sort of someone from the ibm red hat camp to say: okay, get amazon to say, okay and get vmware to say, okay, and I think if that group is willing to say no problem, then cl the sig cloud provider uh has no objection.

A

uh Now, having said that, um you know I would suggest uh nick uh I'm guessing you're, not the best person from amazon. Is this more a key short thing.

B

Yeah, this is definitely kishore's uh preview.

A

um So yeah sorry, I'm just going to try and find you the right person to chat with bridget stephen who's. The best person on the vmware side.

F

I think it's andrew.

A

Okay yeah, so if you can get andrew to sign up for vmware and uh el mico, do you know who the right person started for the red hat? Ibm side would be yeah that'd, be richard tyce, okay, so sorry to give you homework bridgette. But uh those are your three names. If you can get those three people to sign off, then I think uh sig cloud provider has signed off for you.

C

Awesome um I appreciate it and this that homework is exactly what I came to this meeting to ask about. In this context,.

A

Awesome all right all right did anyone else have any topics they wanted to chat about.

A

uh Then I think we're going to go to uh the major bugs uh is yeah peter's still here. So one thing I will mention peter uh step one. I apparently click press the wrong link. um I gotta fix the bugs club.

A

Should probably fix the bug triage to cover cloud provider, but until I do um you're, probably better off filing against kk than the cloud provider repo.

A

Just just to mention that, because I I don't want stuff to get lost uh all right.

A

F

A

All right, sorry, so far, this is not looking like a cloud provider more like a node problem, but maybe I'm missing something.

A

Is anyone seeing anything that suggests to them that this is a cloud provider problem.

A

All right, um then, I am going to go ahead and uh signaled this one.

E

I was gonna say it sounds like something kublet related.

A

Not sure it's entirely cubelet uh note problem detectors usually run as its own service um and then there's the underlying uh vm image, but all of that still tends to be owned by node, okay, cool.

A

uh Oh, is it it's remove dash sig, isn't it and then.

A

No, I got that wrong, damn it nope. I got it right: okay, cool.

A

A

This seems to have your name written all over it.

C

um Is this I feel like I gotta, look and see if this is the one I thought it was. Is this the one that I was asking about yeah? This is the one that we mentioned that we talked about earlier, which is to say there is a pull request to fix it, that we need to that. I attached you to okay. If you click on the pull request to fix it.

A

um So, in fact, what I'm going to suggest then is they've already got that we've. If we've already got a fix and that's who's fixing it, then I'm just going to assign the bug to them. Wonderful.

C

And yeah I assigned you on that um that pull request, because you said that you needed to oh.

G

C

Absolutely yeah: you said that you were somebody who could review that yeah.

A

I will uh that that will probably happen tonight. Wonderful, thank you all right um and joe you're already assigned on this one. So uh I'm assuming that we're the right things are happening and we're just waiting for bowie.

D

I'll circle back on it make sure we're good.

A

Awesome, I think that is everything we have so, uh if everyone's good with it, I can give everyone 20 minutes back.

E

Great sounds good yeah.

C

I guess I just have the question: this is the question I've been asking in every meeting, which is hey two weeks from now. I'm guessing we're not having this meeting on account of it's the middle of kubecon, but I could be wrong.

A

We are not having well, I will put it slightly differently. We might have this meeting, but if we're having this meeting, it's going to be outdoors uh with the people who are attending cubecon.

C

Just putting that out there so that you might want to put something out for the canceling, the individual october 13th meeting. If you want to.

A

Much appreciate it that isn't.

C

A

That is a very good reminder. Thank you, but on that topic I will actually suggest that for those of us who are actually attending, uh I would actually like to uh gather in person uh it's kind of hard to remember what cubecom was like in the pre-covet days. But uh this is where we build our community and building trust and learning to work with each other is integral to making all of this work. So I I would very much like to get everyone together.

A

I might even be able to swing a small budget and go have a an outdoor dinner or something so please reach out if you're interested. Thank you. Everyone bye, bye,.