Kubernetes Cloud Provider Special Interest Group, 28 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cloud Provider 2021-04-28

Description

Discussed the PersistentVolumeLabel Admission Controller ; Discussion will continue in the next extraction meeting.

A

All right good thing, I'm recording yes, okay, um hi folks uh today is wednesday april 28th. This is the bi-weekly cloud provider meeting and uh yeah we'll just get started here. Let me okay I'll wait for that to.

A

Settle um yeah, let's just uh I don't see anything on the agenda. So if folks want to um add items, uh I wasn't here the last thing because I was on vacation, um but I did see some topics that were discussed. I don't know if there are any follow-up items but um happy to to uh follow up on those as well, um but yeah. Let's, let's go straight into the provider updates, um I think uh given alibaba and baidu are not here. uh Aws is up.

A

First, uh I don't see nick, I see kishore on the call you want to give an update.

B

Yeah, except I don't have much updates on the aws side.

A

Okay, cool, I don't see azure folks, google, any updates. I see jared's adding stuff yeah. We are in progress.

C

Of moving the entry code into the separate repository okay.

A

All right uh idea: machine updates.

A

uh I don't see kendall here for openstack on on the vsphere side, um we need to cut our 120 release, um which is which was blocked on that cherry pick pr from from cc uh and as soon as the release is up for that one. We're gonna we're gonna update our vendor, at least uh for four kk and then release 120.. um That's that's about it from our side extraction, migration. I see someone added this topic.

A

Yes, we do need to do this.

D

So we actually had our initial discussion uh in the last uh extraction meeting. uh I will be putting the uh meeting uh up on youtube so for at least steve nick and andrew who weren't able to attend. uh We we had the meeting. We did an initial discussion, which was mostly outlining the problem and um sort of outlining what the possible solutions were.

D

I think we came up with uh two fundamentally two solutions, um and so I one we will have we we're gonna have another re, react the meeting at the next extraction meeting and try and come to the decision, but myself, michelle nick, not nick kishore. I think actually, my apologies uh all went over the possible solutions and I think there was sort of a divided opinion. Some people preferred the web hook solutions. Some people preferred more of a an operator solution.

D

um There are pros and cons each way, so we we just decided to let everyone discuss it, think about it and then come back and we would try to resolve to a decision next time, but with various folks, not being there um and dims not being there. We decided that it was better to just publish the meeting and then come back on the next one, so I should be making sure that that meeting makes it up to youtube this weekend.

A

Okay, I think the for this one uh I mean: do we even need labels on tvs anymore, because I thought that some of the topology information was being pushed into actual fields in in pvs.

A

So could we just maybe this is impossible? Can we just push users to use those or or at least have our controllers and csi consume those instead.

D

Of labels, so my understanding is that the where this kicks in and is important is, if you want to bring your own heart is: is it sort of in the bring your own hardware scenarios where you're like?

D

I have this cluster, but I have a pre-existing storage device that I would like to bring in uh and that may just have been like a cloud provider, specific uh persistence uh prior to your kubernetes cluster, but for those scenarios you still need this controller uh and just as a way of getting them to attach, depending on which cloud provider you're in so we need like. So essentially it is for those scenarios. We need a cloud provider specific solution uh and then there was split opinion.

D

um Some people preferred, like I said it's kind of an edge case. um It is only needed for legacy storage, but assuming that the cloud providers want to support, you know attaching legacy storage to new new kubernetes clusters, uh then then we will need a solution and there there seem to be two basic ways we can go about. This.

D

uh Did that answer your question.

A

Andrew, uh I I think part of it yeah. um I think the part I'm confused about is you mentioned like bare metal, or uh you know, people bringing their own.

D

Yeah I mean bare metal is one scenario, but I think even in like sorry, I'll use the gke example. If you have a gce pd and you want to bring from from your gcp project or what, wherever you add it and you want to attach it to a gke cluster.

D

I believe you need this for that. I think aws ecb uh going into the their store uh their managed kubernetes has similar issues. So that's sort of the scenario where we believe we need it. Okay,.

A

And specifically, the labels needed for, I guess, scheduling, like easy scheduling and I.

D

A

D

The attachment but yeah okay, um I'm deferring to michelle on this, but I I I have very little problem deferring to michelle on storage issues. She is very concrete. um I would suggest that uh these are all great questions, but probably make more sense to bring them to the next extraction meeting.

A

Okay, yeah yeah like we can, we can talk a little more detail there. I think the only thing I'm confused about is the fact that the the persistent volume label at mission controller um is like it has a built-in logic for the five like entry cloud providers. So it's it's not actually a generic. um I'm sure you know sorry, I'm just kind of talking a lot at this point, but like it's a it's, not a generic admission controller, it's it operates absolutely.

D

Yeah, so so I I think the the the general vision is either this becomes a a uh a cloud provider specific web hook or it becomes a cloud provider specific controller uh and that we could try and generate a certain amount of the code that is consistent across kubernetes clusters, but that, in fact it would be a cloud provider, specific piece and so we're just sort of now we're just discussing which cloud provider specific path. Do we want to take gotcha? Okay, that makes sense.

A

Okay, cool yeah: let's chat more on the next call anything else on uh extraction migration right.

A

Okay, uh all right, so I don't say anything else on the agenda: did we want to touch on uh 1.22 caps or any of the previous topics from last week?

A

Or do you all just want to go straight into bug, triage.

B

uh So I just want.

C

B

The uh service reconciler issue that I brought up last meeting uh thanks andrew for jumping in I haven't, had a chance to like respond back yet, but we are having some discussions on this uh issue so.

A

Okay, I think I think um so the the description of the issue or the race condition makes sense to me. I'm wondering um how the issue actually like manifests into problems on like on on eks or whatever platform that is seeing the problem like are you? Are you ending up with like stale or bouncers or like.

B

Just what yeah, what I'm saying is like not the still load balancer, because delete gets called eventually that will remove it. What I'm seeing is like the security group rules are getting leaked and not clean up properly and they build up over time and that's causing a nuisance uh in some instances. So uh so what I saw was like when we delete remove the rules, there's another thread that runs concurrently. That adds the rule back.

B

So that is what I was trying to avoid. So what I was hoping was if we could queue up the service, all the into one reconciler queue uh at least like for one service. We only have one operation at any time. If we can guarantee that somehow this issue should not happen, I mean we could fix in the cloud provider code, but it just becomes ugly and, like all the cloud providers do, the same fix uh would rather want to invest some time to get it right in the service controller.

B

That was my thought process on this one.

A

Yeah, I agree with you. I I think the so my my gut feeling is that the right fixes um what you mentioned is um at a like a work queue where we like. You know uh pop off the work queue, and only you know, one one instance of reconcile happens at the same time and then for the for the update case, probably at the beginning of every reconcile.

A

We should check if the service has a finalizer and if it has, uh if it has the finalizer or sorry, not the finalizer, if it has a deletion, timestamp, meaning like it's supposed to be deleted, um but for some reason like there's an update that runs, we just skip the reconcile and, assuming that it's trying to be deleted.

B

Actually like that, may not be foolproof, because it just depends on how the race condition manifests like it could be that uh after the deletion uh like update, is running and then deletion time stamp gets added later. So we never really know like uh that. That may not be like extremely uh safe way to right, but if, if.

A

Deletion time stamp was added afterwards, then the delete should have ran right, like the delete operation would happen at some point after which would clean.

B

No, that's correct, that's correct but, like say obvious, trigger trigger delete gets triggered, but then those operation will uh like call the cloud provider api to actually delete resources. So that will take time. So we can't guarantee completion right, so there's no way that we can say okay. This is going to complete in this money this much of time. So, while the update is in process and while the delete is in process, they can run into each other.

A

B

C

B

Like if we could just run one operation for any resource for for a given service resource, then we're good, even if update gets invoked after delete is complete. That's fine, because we will not see the resource anymore, so we can say: okay, it doesn't exist. We will you can just get away cleanly. In that case,.

A

Which by resources, do giving the service or not or like the cloud resource.

B

The service resource, I'm talking about the service.

A

Because my thinking is that, um like you, could still run into a scenario where you do the delete um and an update runs afterwards, because there's still going to be delay between when you remove the finalizer and when the service object is completely deleted and like during the delete, there could still be an update that happens and queues up uh like adds accuse of another task in the work queue in the meantime, even if the service was deleted in between the time the the task was added to the queue when the finalizer was removed, I'm not exactly sure if that would fix it, but I I mean, I think we I think we can.

A

um We could probably open a pr and like just go. We can probably just uh run with your suggestion and then maybe we can test to release to figure out um if that behavior is what we need.

B

Sure that's what walter suggested as well like last meeting that uh bring it up and see if it's okay, to open a pr on this, I can definitely go ahead and do that.

D

I I'm also just going to mention that I mean, depending on the controllers, you're talking about um updates and deletes, are always going to be able to run concurrently and there's no way of avoiding them, since they may not even be running in the same process.

D

I mean a classic case being uh the update is being done in the in either the kcm or the ccm and the delete is happening in the other and that that is just the sort of thing we have to be able to handle.

B

Sure I mean if they are distributed. Of course we will have to build the safeguards, but in this case it's a single controller right. So if oh.

D

And we should make sure the controller. The controller doesn't conflict with itself, but I'm saying, while we're doing these, these are not these. These are not resources that are exclusive to this controller. So just be aware that you know there will be service, controller and other things playing with the service right.

B

Yeah, I totally understand.

B

A

A

So yeah I mean let's uh yeah, let's pick whatever you think is gonna work best and we can continue the discussion in the pr.

A

uh Did you want to follow up with your or did you want someone else to take it.

B

I can take it I'll, be able to put some time next week uh this week, I'm fairly busy, so I can definitely take it up.

A

Okay, cool thanks.

A

All right, I don't see anything else on the agenda. Did folks have other topics. Do you want to go over bug scrub all right? What was where did we leave off from last.

A

A

Are we just going through the needs triage at this point.

D

uh Looking through it, I'm the last one I see newest was five six, seven, eight seven.

D

Right unless we forgot.

A

To I, I think we I think we went through.

A

I think we went through all the the issues that didn't have a triage label like the really old ones.

A

Okay, that'd be awesome, and I I don't know that's what that was my um recollection last time and then now we're just going now where we we we're going through the top of the list again based on the ones that do have the needs triage.

A

A

um Sounds good, let's do it, okay, are we going from uh latest oldest or do we.

D

Want to as long as we remove the triage label it, I don't think it makes a huge difference. These ones, it's not like yeah. As long as we remove the triage level we're good okay, I.

A

See there's only okay, there's 28, that's not bad all right, maybe okay, all right.

A

Okay, multiple, you guys.

A

Okay, I see I'm assigned to this, I'm pretty sure there was a there's, a pr open. uh This.

A

B

Sources and check this is little. uh I want to take a look at it as well: uh nine, zero, two zero okay.

A

Yeah, I think this music- yes, I recall this was um related to like dual stack on aws. um I don't know if it blocks dual stack, but um I'd be curious. Like does dual staff work on aws.

B

I mean stack yeah, it's not supported uh for the entry controller. We'd have to use the load, balancer controller for dual stack.

A

Okay gotcha, but this.

B

Will take a look.

A

Dual stack nodes, though, like for having the nodes, have dual stack, addresses.

B

um I'm not certain I need to look at so I can go over it and see. What is the issue.

A

All right failed amount due to okay. This is a storage problem.

D

Sorry, just quickly did we assign kishore to both the uh reviewing the fix and the the bug? No, not the fix.

D

All right, sorry about that! No.

A

No good call, okay, all right this one, uh I'm going to remove.

A

It seems like the azure folks are already trying this one. So I'm just gonna triage accept it uh all right.

A

uh Custom gls policies for aws clb.

A

Can I keep short, can I send this one to you as well.

B

Yeah, we're not support supporting any new features here so.

A

Yeah yeah, I'm kind of assigning it to you too.

B

I'll do the uh needful.

A

Thanks um and yeah you can, I think, yeah you can probably just close it um don't even have to okay. This is a cube up one. I think.

D

Yeah, I think this one should go to me.

C

Can you see me on that? I want to decision.

C

A

C

A

Yes, unable to update low bouncers target group duplicate.

B

Is it aws yeah I signed to me I'll take what.

E

D

Are you making sure to remove the triage label.

A

A

Okay, I guess these are just accepted, then.

A

Yep, hmm okay,.

A

This is another aws one. Do we need security, health check.

B

uh Yes, I mean this is expected because we create uh health check rules uh for the number of subnets or the availability zones that are there. uh So in the entry controller we may not have much fix for now. Unless nlp supports a security group, uh it should be the way you can assign to me. I will reply back further.

D

Yeah, I mean definitely that that explanation belongs in this and that's awesome. Thank you, sir.

A

Okay, this is a vsphere issue, so.

A

I will self-assign.

A

Or you know what I'll just assign this to someone on the team.

A

Should I be so, am I supposed to triage accept them if I'm not sure if their actual feature requests.

D

Or bugs I don't think it matters all we're saying with triage accepted. Is we've assigned someone to look at it? Okay, that's fair! All.

E

A

Where was I right here, you know what I'm just going to open a bunch of.

A

A

Okay, let's do these and then maybe we'll call it all right uh use of. Oh, this is a kermit okay, I'm just gonna triage except this one. This is just uh like an improvement.

E

Yeah we'll be able to find a reviewer if we need more.

E

A

uh Cast ingress using nlp has known how talk, for instance, after ppc and submit expansion.

B

Yeah, it comes to me.

B

Yeah this one as well.

A

Okay, I'll just make it more practical.

A

A

The gce scripts, I think so.

A

D

Yeah that works.

A

All right, uh you can't still don't service stuff updating. You know this video, but.

B

Yeah this is likely because of the race condition. I will take a look.

A

B

Sometimes it takes like 10 minutes or more. It just keeps on trying to update it. I will see okay.

A

As cross component.

A

Yeah dc, do you know for review or someone else uh is time to run this.

D

Does it make sense to assign this one to you, joe.

E

What do we mean by cross component in this.

E

E

Cc I mean for now.

E

B

You can see me.

E

I mean certain like the gcp cloud controller. We're definitely going to do that, but I'm not sure if this is like referring to just like, like the test ccm or what it means.

A

Yeah, I think the so I think this came out of the fact that there was that bug that um that cc fixed where the cloud provider.

D

Flags yeah, the wiring of the parameters, wasn't working.

E

ah I'm thinking of something different, okay, yeah, that makes sense. Okay, we'll follow it.

A

Yeah, this is just saying, like integration tests to uh just make sure the place, the basic plumbing of ccm is working.

E

Yeah yeah, when I was thinking across component, I was thinking about something else I was on my mind. This is much easier: okay, cool.

C

I think we have another issue to track the um umbrella test framework for collaborator or ccm.

C

Sorry, this yeah sorry, the previous one, just maybe just for integration test, and we have another um issue for, like other tests plan yeah.

E

C

A

B

A

B

To you sure, yes, please and lv doesn't have sg, I'm curious, but I'll take a.

E

A

Performance test, failure for openstack.

A

I'll assign kendall she's not.

A

A

Right incorrect notepad chosen for ssl house.

A

B

Yeah assigned to me please I'll, prepare.

E

A

Let's see where did we end off? Do we want to do more, or do we want to call it today? Let's see there's about.

D

We can call it, and the nice thing is as long as we triage accepted. We don't have to even worry about tracking, where we were.

A

Yes, okay, so yeah I refreshed it and I see uh I see about 14 left, which is pretty good, so yeah. Maybe we can probably clear these out the next run. I think this one we can we can. We can triage this hold on.

A

uh This one is the one we just talked about, and we can uh this one.

A

Okay, cool yeah, let's uh yeah, okay and then we'll we'll revisit the last the rest of them next.

D

Time all right, cool.

A

Good awesome all right anything else: books, wanna, discuss.

A

All right cool thanks all see you in two weeks.

B

Thanks everybody bye.