Kubernetes Cloud Provider Special Interest Group, 3 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cloud Provider 2022-08-03

Description

Agenda: https://docs.google.com/document/d/1OZE-ub-v6B8y-GuaWejL-vU_f9jsjBbrim4LtTfxssw/

Discuss PRs merged/need attention for 1.25
https://github.com/kubernetes/kubernetes/issues/111029
Implemented in:
https://github.com/kubernetes/kubernetes/pull/111033
https://github.com/kubernetes/kubernetes/pull/111462
https://github.com/kubernetes/kubernetes/pull/111466
https://github.com/kubernetes/kubernetes/pull/110066
https://github.com/kubernetes/kubernetes/pull/108838

https://testgrid.k8s.io/google-gce#disable-cloud-provider

“kubelets specifying --cloud-provider=external will add a taint node.cloudprovider.kubernetes.io/uninitialized with an effect NoSchedule during initialization. CloudNodeController will get NodeMetadata from cloud provider, add LabelInstanceTypeStable, LabelTopologyRegion, LabelTopologyZone labels, finally remove uninitialized taints.
if cloud provider has some additional work for initialize node, before works done, want keep this uninitialized taints, can add some hook that CloudNodeController get node initializing information from cloud provider, if not completed, not remove this taints?”

A

All right, hello, everyone and welcome to the August 3rd meeting of Saint Cloud provider.

A

um Let's see please uh remember to mind the code of conduct and we'll go ahead and get started with the agenda or actually take that back, we'll go ahead and get started with um triage.

A

So let's take a look at our open issues. Right now, we've got. This looks like about five.

A

um First one is AWS NLB.

A

That doesn't sound great.

A

A

Node State changes causes excessive lb reconfigurations.

A

This is interesting.

B

But the cloud provider specific or as a general issue.

A

Good question: uh they seem to think it's.

C

I think it's a general issue, but it's a general issue. In a controller we own.

A

Yeah, that's what that's! What they're saying.

A

It looks like it would be because of the service controller, I guess.

B

um I'm, why are we seeing that we are owning the service controller, I.

C

C

I didn't take them to think it was the service controller. I thought they were saying. It was the cloud life cycle controller is.

A

A

C

Or I mean they're definitely talking about Services as well I thought I. Could you go up just a little bit I? Think? Oh no!.

A

A

So it's Cloud API calls for lb reconfigurations.

A

um I know the the reconfigurations would be directly because of the service controller right, yeah.

C

No, that doesn't make sense.

A

C

Yeah I mean the funny thing. Is they keep referring to the CCM, but it's there at which I think it's less about the CCM? And it's really what matters is which controller within the CCM right.

B

Yeah that probably work around just around the service controller from the KCM wouldn't help here. So it's not CCM specific foreign.

A

So this part here, like canceling, the scheduled eviction for the endpoint, hosted on node 25 for service. One.

A

um Is that KCM, like.

C

And well, I mean part. This is part of the problem. Here is they're they're talking about controller managers, but to understand this problem we need to understand controllers, not controller managers.

A

Right and that's what I'm trying to do right now and I'm I'm wondering like pinpoints, is its own controller right. If there's like a.

A

Or something yeah so probably spread across multiple.

C

Yeah, it's just when I read things like the vision I have is to remove the sync function for transitioning node Readiness from the CCM altogether, which is like maybe, but it's not about the CCM. It's about the individual controllers and in fact, which controllers run and which controller managers uh actually changes by cloud provider.

C

So we really really needs to be a discussion about controllers, but.

A

Yeah, well definitely should be I think we can definitely.

A

uh Accept it at least um to discuss it. um Yeah I, I won't assign it yet, but I will CC all.

C

Of us actually I I would suggest we not triage it. Yet, if we're not going to assign it okay, fair I, think she's, seeing it's a good idea, but let's leave it open and triage it once we feel like we have uh a plan.

C

A

B

A

Jpo second,.

B

A

A

All right who do you want on this one.

C

You want this, so you want to give this one to me. Yakim.

B

Well, let's assign me probably I would think it what to do exactly with this. But let's assign me.

B

A

C

I do know that we would just recently went through about of upgrading the cause version across, so it may be worth checking to see what the actual cause version was or is for these tests.

A

Azure Bridget.

A

Is not here, yes,.

C

But we probably what I would do is at least Cc or assign it to Bridget and she'll find the owner, although if uh feisker is on it, that's probably the right owner.

A

Yeah I feel like we can just accept the triage and leave it at that, like they they've they've got. This looks like they've got this yep.

A

C

Right is it a science device gear, though.

C

Yeah, let's just decide.

A

C

A

To LZ hitching all right.

A

Who hasn't commented on it.

C

Yeah but Vice um so I'm assuming.

A

Okay, I think that's good. Oh.

B

B

C

B

So I needed to add metrics to uh KCM CCM to kind of track. The A3 migration- and this is already merge from the.

B

Extraction migration point of view we are able to to track.

B

C

Also, should we just decide it to you and then you can close it out.

B

Yeah I can write the explanation for this and what was needed and foreign.

A

T updates um looks like not a lot of people are here today, so um can start with AWS I, don't think Alibaba and baidua. Here um I was just going to copy this change and we just had a bug fix for um basically this this rate limiting issue. We had.

A

A

Azure see nobody's here for azure um do either. One of you want to take gcp.

B

Well, from the last update to the changes that we have freezed the 1 24 image and are starting working on 1.5 cool.

A

C

One additional quick Point, um we did upgrade I think a couple of the OS versions. uh It hadn't been done in a while, so yeah you'll find uh like a new version of cause and I think, maybe even new versions of Ubuntu for all the gcp stuff.

B

It's gonna be fun, yeah.

A

Okay, um cool and then extraction migration.

C

And uh so we'll um the test grid is running. It's not pass. It's not even uh properly running yet we're hitting a interesting problem uh with getting the cubelet to actually successfully come up. The cubelets crash looping on uh a certain uh a cert signing issue uh so I'm reaching out to the security team. uh Probably you know Mike and trying to work out how to get past that um it's a little tricky it's, but we solved the problem on cloud provider gcp, but I'm trying to work out how to get past it on KK.

A

C

Because right now we get past it using a Google binary that isn't available on KK.

A

A

All right uh should we move on to the agenda. Actually, if we could take a quick break here, um so two people from my team I see have joined uh Earnest and receita. Do you guys want to introduce yourself introduce yourself sure, hey I'm, Ernest, uh just joined AWS two weeks ago or one week ago uh before that I was actually working on azure uh related to kubernetes so nice to meet you all.

C

Congratulations.

B

Nice to meet you thanks.

A

Cool thanks for joining.

C

A

Excited to be contributing to the Upstream issues as well.

C

Yeah, congratulations good to have you on board.

B

C

B

Nice to meet you thanks.

A

For joining guys, um all right and now agenda, so I wanted to just go over uh like 120 25. What we, what we got in um if there's anything, we need to I know code freezes yesterday, but if there's anything, we need to uh try to merge last minute or get an exception for it. So I have a couple uh Jacob you want to start with the um the metrics thing. Is there anything else you want to mention about that or.

B

Well sure uh I've merged metrics required to track if it Converse are running in KCM and CCM running manage controllers, it's a single metric that provides one or zero for service road note, ipam uh and other sharp controls between the KCM and CCM, uh and it's either one or zero, depending if this is running in in specific control manager. uh It's especially useful when someone is working on aha migration to kind of track. The success of the of the migration.

A

Got it is it, can you remind me, is it, um but you said it's one metric if it's one or zero there's.

B

A One controller, it's one that is running manage instance. It has labeled per controller and the value of this uh metric would specific label is always one of zero. So there is label for service controller. There is labor for uh Road controller and so on. Yes,.

A

Cool um all right: do you want to link that here sure.

B

Probably the issue will be the best part for for that.

A

All right, so, let's start with this one.

A

A

Water I feel like this could probably just be merged, has lgtm. um If you want to approve this, I can explain what it is um and assuming you agree with it, um but it's just a.

A

Replacement, let me just change this really quick to.

A

Oops where's the video thing. Now there we go.

A

um It's a replacement for like a long-term replacement for the cloud provider flag, um so there's a deprecation warning on it. um My initial PR here was to just undeprecate the flag, um but I got pushback that we don't need a flag with.

A

You know like arbitrary values anymore, because the only thing that we need to um uh signify- or the only thing we need to like um allow users to configure, is whether there's an external cloud provider or not, um and the reason we need that that ability is because we need to be able to apply the um the taints or not apply the saints that the cloud provider will remove.

A

um So the suggestion was to add another cubelet config, which is just like a Boolean. um So this is kind of supposed to be the long-term configuration option and then the cloud provider flag will be deprecated um and yeah. That's all this does.

C

Yeah, that sounds fun, um yeah I, like moving away from the cloud provider and then.

A

Yeah, just a simple, simple, config option.

C

All right, then, just to throw I I like it I'm happy to approve it. I I do feel the need to ask one question which relates to a conversation we had in cloud provider extraction mm-hmm.

C

So one of the things in cloud provider extraction that came up in the last cloud provider extraction meeting, but we didn't get to the bottom of- was that one of the new Cloud providers was was wondering if there was a nice way to hook in to be able to run extra controllers, and they were having a problem doing this, because the existing um cloud cloud lifecycle controller is going to be clear. The the tanks and what they wanted to do is do uh extra configurations, so I mean there.

C

There seemed to be at some level three ways to go about this one. We tell them to patch that that that controller, but that means that they have to carry patches every time and we've always tried to avoid peop telling people to carry patches, because if we change the business logic that can really make carrying those patches, painful and expensive, we could add a hook into the code. So that all they have to do is implement the method and that method is called it. A well understood point in that controller.

C

So now you know it's it's a lot easier and you're not really carrying a patch, but someone's got to design that system. The third option would seem to be if we could find a way to add an additional taint, and so what I would the question I'm going to ask is, and we could even do it with an extra flag is, would it make sense to sort of re redo this instead of it being you know here, you know, do you want me to add the standard taint is yo. What are the taints?

C

You want me to add, and then the default value would be this the default cloud provider chain, but you could actually provide a list of other tanks as well. If you were going to run extra controllers.

A

Yeah, that is, that is uh interesting use case and it does seem relevant.

A

Oh okay, yeah I'm, convinced that we should consider that at least before merging this, if you want to comment.

C

On, let me just quickly pull up the the use case.

C

C

All right I will find it later and bring it back. I'm not I'm, not able to find it off the.

C

Yeah I'm gonna need actually hang on. Let me try one other place.

A

C

Oh I see I, didn't get it as a bug all right. uh Let me just add it to the dock. Then.

A

Okay, do you mind.

C

A

Ccu on this.

C

Yeah no I don't mind at all um I'm just gonna add the quote:.

C

All right, yeah I, just added it to the agenda. If you want to take a quick look at the agenda, so this is actually just a question. I got.

A

Yeah I mean it makes a lot of sense. um Yep, so yeah um I could see.

A

I could see us taking that into account when we, you know, if we're gonna change the configuration API we might as well take this into account so good timing on their part.

A

um Okay, yeah I C do on it. uh Neither one of us can kind of look at this, and and summarize it on the on the on my PR, and we can I, don't know, think think about what we want. Yeah, very cool, all right last, one that I have is what book framework thingy. um So I didn't get to this until like this week because of some prior uh commitments, but um it's not super far away, but I would need probably like Walter uh your um commitment this week for next couple days.

A

So if, if I have that I think we should try to file an exception, um what do you think Walter.

C

Yeah I I'm, all for it, if you could ping me on slack I, will make sure to devote some time to reviewing this cool.

A

Will do um yeah all right I will I will do that I'll paint on slack and I will ping you about the exception. Getting that um in motion uh all right, cool.

A

Do we have anything else that we want to discuss.

A

That's an agenda.

A

Anybody cool all right thanks everybody.

A

Thank you see you thanks.