Kubernetes Cloud Provider Special Interest Group, 25 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cloud Provider 2022-05-25

Description

Agenda: https://docs.google.com/document/d/1OZE-ub-v6B8y-GuaWejL-vU_f9jsjBbrim4LtTfxssw/

[Joel] AWS bug with custom DHCP parameters
https://github.com/kubernetes/cloud-provider-aws/issues/384

Kubecon recap

Should cloud providers enable –external-cloud-volume-plugin be enabled for all cloud providers?

Next kubecon - meetup as sig?

A

Hello, everybody welcome to the sig cloud fighter. uh Meeting wednesday may 25th. um We will be going ahead and getting started.

A

So we're going to go ahead and start with triage and.

A

It looks like we have three issues.

A

um Some of these look a little bit familiar.

A

Amount device failed for volume.

B

It looks like they say: they're using aws ebs entry, dynamic provisioning down at the bottom of that section. I don't know if that.

A

That is relevant. Certainly, uh where do you see that.

B

uh Right above the, what did you expect to happen.

A

uh Perfect all right, um let's go ahead and accept this.

B

A

I can assign myself for now.

A

This one also looks relevant to me.

B

Yeah I read this one and I thought it was interesting because I wonder I'm not sure, but I wonder, is it possible to make some of that stuff configurable because, like they're noticing that there's a bunch of traffic that they don't want in their logs or whatever, but like? I wonder how configurable that is.

B

I don't know if that's possible, but maybe that's the sort of thing that would be valuable to make configurable.

B

That sure is chatty.

A

Yeah and it's actually they're running into bandwidth issues from this or.

A

Yeah, I think they're they're.

B

Claiming that there's reit, limiting that, I guess this is getting in the way of traffic they care about or something.

A

I hear a trivial optimization.

A

I am interested to see what actual negative consequences they're experiencing from this, because.

B

Yeah, I've never.

A

Heard of someone.

B

Yeah, I'm just wondering what metadervis, what the metadata service rate limiting um manifested still itself, as you know like how do they see that it interesting.

B

Oh, that person says that has actually changed recently interesting.

A

I think he's just referring to the um extraction to external ccm, so I think this is referring to cubelet. Hence the hitting the metadata endpoint, um of course uh go ahead and accept it and dig in a little bit more see what the issue is, and then is this the one from before.

A

B

This one has been around for a little bit. I actually looked at this one just because we just um put the yeah I see uh so we just put um the uh service selector um to you know like.

B

Basically, this person is saying we think that this is solved and then I dug in and I saw that they um are running 1.19 so like I have no idea if they've managed to reproduce it recently, yeah.

A

B

I'm not I'm not sure if we can give them any solutions if they can't test with a more recent version. When you see.

A

B

You dig into the specific version that they're experiencing this with.

A

B

And then you can see the comment from my colleague from april 22nd, where he's like. Well, it's fixed and master. It's not released yet so.

A

Yeah, okay, would you mind at least triaging this and determining if we should investigate further or if it's already fixed.

B

Yeah just go ahead and trash accept it. I want to say this is already assigned to me, but maybe it's not yeah.

A

B

Is I don't know why it still says? Tr needs trash, sorry about that.

A

No worries, I didn't accept it because I wasn't sure if the, if you wanted it so um okay.

B

It's just like, I don't basically like if they're running 1.19 and this might be fixed currently, then, if they can't replicate it currently, I'm not sure. If there's a lot, we can do to help since we're not going to fix 1.19 yeah.

A

Yeah: okay, cool, okay, cool.

A

All right, let's go ahead and go through subproject updates, so I will just go actually I'll give everyone a minute to fill out any updates that they have um and then we'll we'll go through the list.

B

I just put the link to the google doc in the chat in case people didn't have it handy.

A

All right um looks like it misses first, uh so we have some new releases of the ccm um they're, all stable versions. Now, so we had them tagged with alpha for quite a while, but we're stable and then tagging controller uh support was merged. That's a controller that we were working on um that allows uh specific tags to be added to instances that are specified um in a config file or in a flag.

A

B

Yeah, um I mean obviously that was just kubecon, but the uh one thing that I noted that recently happened is we just got a release out that had of kubernetes patch releases that actually has a bug fix that affects entry for azure file, so that might be of interest to anyone paying attention to that um other than that same old, same old.

A

C

uh Hi uh we are preparing repository to cut release 123 to the clean cut on it. It requires some library updates.

A

Got it ibm, no updates, yeah yeah, no updates for me right.

A

Yeah, okay, uh I don't see walter, um I don't believe we had an extraction migration meeting last week. I think some of us were at kubecon, so um I didn't even look to see if if it was happening well,.

C

I I was during that meeting enough on show up other dummy and walter, so we just discussed our internal google.

A

Got it got it? Okay, um cool, so I think we can say no updates for that.

A

um Moving on to the agenda job.

C

Hi uh so added this couple of weeks ago, and there has been a couple of responses since on the issue, uh and it seems to be that there's it's basically looking at the way that the person's responded it looks like it might be, a no fix, um but I just wanted to quickly discuss it and just explain what we're seeing um and then see what the general consensus is. Whether we should fix this or not so basically, what's happening is.

C

In aws the dhcp options you can set those to have an empty domain name, at which point it starts returning different values for the host name, which cubelet would normally pick up uh versus the um private dns name.

C

And if you set the correct, if you set the domain to be something that is, you know your own domain or whatever it works correctly and those two match up. But if specifically, if it's empty, the private dns name ends up. As with this mismatch and the way that the ccm falls back looking at the node when there's no provider id relies on. This um relies on the private dns name, rather than the actual value that cubelet might get so because the metadata doesn't have access to the private dns name.

C

It's very hard to put anything on cubelet to make this work. The only workaround we found is to try and set the provider id. um If you do the pride id, then this works and it's fine. um You like, we don't even need the name check, but basically there's a there's, a mismatch in how the name is being used to look up the provider id uh and what it the the upshot is. It looks it's very difficult to diagnose because the node gets created by cubelet and then ccm deletes it within a few seconds.

C

So it looks like your node has not joined the cluster correctly, um but yeah, it's just it's very confusing, uh so damn who's written this up is actually much more knowledgeable on this than I am, um and you know I asked him to write quite a lot of things, so he has done um basically the question here is: do we expect that the cube should always be registered with the fqdn, even if someone's saying, custom magic, dhcp options that mean that the fqdn technically shouldn't eat just the standard hosting.

A

um So a couple questions, uh so are you saying this does work if the dhv option set is not empty like if there's an actual domain name there.

C

Yeah, if you put actual domain name, it works fine. If it's empty, it doesn't work.

A

Okay, and so what's the use case for having that empty, because I'm not sure if I've seen that before.

C

uh Yeah good question can't remember that one uh I to be honest. I forgot I put this on the agenda, uh so I'm a little prepared might.

A

No worries uh I'm just wondering like how common this is, because that could um kind of determine what we need to do um or, like you know, if there's, if there's workarounds for the use case or not, but and then the other thing is um you know, node name is sort of moving towards instance id. um So I don't know yeah, I'm just wondering about the use case and like people on this use case. Are they stuck on this?

A

uh Is this like a required configuration, or is this something that we just sort of ran across? um But that being said, um let's see so that it looks like the the failure mode is. The node is unable to join the cluster, so.

C

That's the node joins, but then the ccm deletes it within a few seconds, which then becomes very confusing.

A

Yeah, that's a that's a horrible failure mode. So if this is a legitimate use case, then I would hope we could improve that behavior. You know, even if just making it fail more gracefully um yeah, but without like digging in more it's hard for me to uh comment on exactly like what the behavior should be.

C

Okay, that's cool. I will try and work out that use case question and bring this up again in a few weeks um because yeah I say I forgot- I put this on there, um so I also make sure that damn and the others that are working on this during the pool as well.

A

No worries I mean, I think it's it's clearly worth looking into just to kind of figure out what the use case is and because the failure.

C

A

Is pretty bad, um so I'm gonna go ahead and accept this uh and take a closer look at it um cool thanks for bringing that up, though, cool thank.

C

A

A

All right folks, anybody have any other items that we want to bring up on the agenda. um I think I guess we can do a quick q. Con recap just for fun, um so I will go first and just mention that we we did. We did a talk. um It's just a sig cloud provider update uh not um really a deep dive, but we we went through like what migration is. Why we're doing it? um A few updates on.

A

uh I don't know like what has made it to alpha recently, what has made it to beta and ga recently so later, migration, um keyblade image, credential provider, stuff like that, and then for those uh cloud fighters that submitted slides. We just talked through the slides and did our best to uh either read them or those that had a lot of text. We would just like mention a few things and let the audience read them um and then got a couple of really good questions.

A

um One from matt booth, who I think is some of your colleagues um who asked about whether or not um the cube controller, manager's external cloud volume plug-in, should be kind of I I don't know if it if it should be required for all you know, but basically like he was saying that probably cloud providers should enable that flag when doing volume, migration um just to uh meet the version sku policy.

A

So I think the idea there is just that if you disable the entry plug-ins, then you're not really meeting um version sku policy, uh I think the maybe the counter argument there is that it should be like moving on to csi and doing migrations should be kind of invisible to users. So um as long as you have a csi installed like that, the api objects um don't stop working, but it just requires a customer to install something. So I don't know it's an interesting conversation.

A

I think we should dive into it um a little bit more in the in the sick. I'll put a note here.

B

Yeah that that is really interesting because it kind of comes to the question of what do you want the default expectation or behavior to be like what.

A

B

Expect if they haven't configured something or you don't even know to configure it, you know kind of giving them that happy path of defaults.

A

C

I'm just going to add as well listeners the message it's actually quite bad, if you don't have it enabled and the csi migration goes through, because any volume that was mounted pre, the csi migration cannot be unmounted. The csi can't unmount it if it was mounted by the entry uh is what I was told by the six storage folks.

A

Oh, that is, I did not know that that is that's not the the case that I thought. Okay, that.

B

Sounds like if that is the case, and it sounds like it may be. Then it sounds like we need a very big breaking change. Sort of warning for people.

C

I was told that by six storage- probably about a year ago, um so we've enabled that in openshift and that's kind of like on through all of our migrations, then we're only going to get rid of it like two releases after uh to give sufficient time for those volumes to hopefully have been unmounted.

B

Oh sweet summer, child. You hope somebody has changed something in what six months. Eight months, oh boy,.

C

That's right, I know we reboot stuff, so we have a get out of jail free card. Sorry about the rest of you.

B

I mean it would be lovely if everyone would update their clusters.

B

Yeah I had a conflict at kubecon, so I wasn't able to make it to the same club provider talk. um But uh I'm happy to hear that there were some good questions and I'm eager for the video when it comes up. I guess post it I'll. We'll definitely want to put a link in the notes here and you know put it on slack and whatnot.

C

I was just checking and it it was uh yansha franek who told me that who is one of the tech leads physique storage. So I'm hoping he's right.

C

A

Well, that's super good to know um all right did anyone else want to talk about coupon or anything else.

B

um I will say that, as a contrast to los angeles, we definitely had a lot more end users and just you know, general people who don't necessarily work at cloud providers or vendors. I'm at this kubecon, which makes me hopeful for detroit. It seems, like things might be ramping back up in terms of you know the customers and community members that we can talk to who hey, I kind of enjoyed vendor kubecon, but I feel like that's, not a sustainable path forever, so um it was good to see it being more active.

A

A

And on that note, um we should do something more at the next coupon. Perhaps like uh you know, if people are planning on being there, we should do a more official gathering um like we had a super unofficial aws thing at this one, but we didn't do anything as a sig, so I don't know it might be interesting uh to oh yeah, we've done a dinner in the past. um We could do something like that or we could do like a just. A quick meetup chat, something like that. um So.

B

Yeah I mean I think schedules can be very hectic but definitely getting. um I wonder if uh I know that there were like the project booths again this year. I wonder if there will be any either um pre-day like half half day session rooms that we could get, or maybe rotating time at like a cncf uh booth or something just to kind of have a little gathering area.

B

Or hey uh this year, microsoft sponsored one of those um activation zone. I don't understand what that means, but there was a like driving game video game. Like actual chairs, you could sit and drive on a video game and so hey, you know.

B

Sid cloud providers just get together and hang out at one of the game.

A

Okay, gameplay.

B

It was cool, so it was like a driving game in the expo hall, while one of the expo halls there were two which was very confusing and so half the time. I would not see people who were over in the other one and yeah.

A

Cool yeah we'll have to do something like that next time, um all right, uh we are out of agenda items. Did anybody have anything else or shall we rap.

A

All right, thanks for coming everybody thanks.