Kubernetes Cloud Provider Special Interest Group, 6 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Cloud Provider 2022-07-06

Description

Agenda: https://docs.google.com/document/d/1OZE-ub-v6B8y-GuaWejL-vU_f9jsjBbrim4LtTfxssw/

List of failing tests due to disabling cloud providers:
https://github.com/kubernetes/test-infra/pull/26747

[bridgetkromhout] test triage reminder - see https://github.com/kubernetes/kubernetes/issues/109913#issuecomment-1123973475 for Nick’s comment about breaking it out into multiple issues per-provider

[bridgetkromhout] cherry-pick deadline this week! https://kubernetes.io/releases/patch-releases/#upcoming-monthly-releases

A

All right welcome to the sig cloud provider meeting. uh It is july 6., please be mindful of the cncf coded conduct and we'll go ahead and get started.

A

um Add yourself to the agenda and or the attending list, and please add any agenda items as well as subproject updates. If you have any, we will start with a bug triage.

A

All right, we have spoken about this one before.

B

Yeah I write down that one again and I thought you know we talked about this before, and maybe this one is just a this is not getting fixed in tree. I think that's the most recent thing you just said nick.

A

Yeah, apparently I like attempted to triage, accept it and, oh, I think it's because it's sig note as well, so we're kind of just waiting or sick node to decide what to do with it perhaps um so makes sense. um Yeah it's it's interesting.

A

I don't know I haven't. We haven't really heard back so I I mentioned that it was something that could be optimized and asked you know. Is it worth spending effort like tell me more about um what is is going on and the person never responded, so um it doesn't seem like it's burning them right now, or else they'd probably be more responsive.

A

So I guess we can just kind of sit on it. um I don't know what to do, though, since it's half triage accepted, do we have a procedure here.

C

So sorry, quick question nick um yeah is this something that remains a problem once cloud provider extraction is, uh it is done yeah.

A

That so, as the cloud provider is right now, no um it shouldn't be a problem, or at least it's not guaranteed to be slash likely is not because this is for um it's basically hitting the metadata endpoint to get addresses which, as you know, is done by the um cloud node controller.

A

So you know there's a chance that we do something like azure does and re-architect it to put some demon set on the node, but as of right now likely won't be a problem.

C

So my suggestion and- and you are welcome to do what you want with it, but my suggestion would be to be a bit more explicit and actually say you know. um We believe this problem will go away in a few releases. Once cloud provider extraction is completed, you shouldn't face this problem. If you're using the external, you know the the uh the the aws distribution found in kubernetes sig, slash cloud provider aws and as such, uh unless there's an imminent need, we don't plan on fixing this in trade.

A

Yeah, I'm just going to say something somewhere like given. um This will.

A

Don't know how to spell that word.

A

uh What else are some imminent need? Please.

A

Okay, um cool. That was easy. I think we're done with triage.

A

So I'll go through subprojects, um I don't have anything for aws and I don't think kishore is on uh okay. So let's go to azure.

B

Hey so um the very cool issue that nick opened back in may about cleaning up test failures. uh We did some work on and I put a link to the few that we still have that we need to diagnose and clean up. But that's uh thank you for leading the charge on that and we're gonna keep getting that uh grid. That test grid to green.

A

Awesome awesome that's great to hear yeah. We need to do some work on that as well cool anything else.

B

Nope, nothing else.

A

D

uh As a part of updating repository to 124 version, which is something that should happen soon- uh I've updated documentation about uh pumping process which started some additional scripts that that should help with the process. So it will not be some so problematic. Next time we'll be handling this. This issue.

D

That's all cool. Thank you.

A

All right and then extraction migration. Walter did you add this one.

C

I did um I need to actually fix that pr, but uh I I have been promising to create that pr for a while, so I went ahead and actually created it, uh it's essentially adding a so let me step back sorry. So there are two feature um flags that we have to test: what happens when you've uh explicitly have uh cloud providers set to external or to force you to show what happens when you have it set to external and right now we have no nothing in test grid or anywhere else.

C

That is actually testing uh what happens on all the tests uh in kkk when uh those feature flags are turned off so uh entry plug-in.

C

That seems wrong. Okay, let me go back. I need to it's disabled cloud provider. I thought there was a different name all right anyway,.

A

What about disabled credential providers.

C

Yeah, I apparently had a copy and paste error. I needed to fix it anyway. You are correct. Yes, the second one should be disable uh credential provider, but um the idea is, we will uh shortly have a test grid that will actually show us all of the tests that should be failing under that circumstance.

C

So um hopefully, I'll have that running this week and we will see just how many tests are failing. um Just so we're very clear about what the expectations are. uh It's not necessarily that all of the tests, all the tests, have to be fixed.

C

The first thing we need to do is get that list of tests and triage them, so some of those tests uh will probably need to be moved to uh sort of become distro level tests, so, whether it's cloud provider, aws cloud provider, gcp cloud provider, azure cloud provider, alibaba whatever that's fine, uh some of the tests uh we may decide, we just don't need and the rest of the tests uh will have to be rewritten so that they don't depend on the cloud provider.

C

As far as I can see, those are essentially our three options. uh All those tests pass today, but they all pass with the cloud providers. So, uh presumably, if turning cloud provider off, fails that then it's uh yes very strong campaigns, uh it is. It is because we turn those them off, and so we need to work out what to do.

A

Yeah makes sense: okay, cool.

A

I'll definitely follow that one. So once we have the list of failed tests like what is our, uh how are we gonna get uh action based on that list or get people to look at it.

C

So my general feeling is that we'll we as a sig will need to send out um a mail that, ex with the list of tests uh explaining the problem- and I my thought is we'll start by sending it to sig release and sig architecture, and we need to basically get an agreement on next steps. uh My my suggestion on the next steps is going to be that we send uh those a a shortened version of those tests.

C

Those test lists to the owning cigs, so we'll go to signal and say: okay, you have here are the 16 tests. You.

C

That are currently depending on cloud provider. um Our suggestion would be one of these three things. You're welcome to come up with something we missed, uh but how would you like them dealt with and that it it's got to be an agreement from the owning sig for each test on how they want them dealt with.

C

um This is then going to dovetail back to sig release, assuming that some people want to run at the distro level, because right now we really on may be suggesting, because right now there is practically no way to uh run tests at the distro level, uh based on an upcoming kk release right so and and just to be clear what I mean by that, uh if we go back to jacob's uh last note on the changes that jacob's making um he's talking about a set of scripts, that will allow his team to automatically update to 124.

C

Now and that's great for that distro, but it sort of shows you how far behind I think gcp and in practice my guess is every one of the other dis cloud provider. Distros is on what I'm going to call the kernel, which is the code, that's actually in kkk, and so, if what we're trying to do is track, um whether or not a recent kk change has broken some cloud provider dependency for testing.

C

We need to be able to almost immediately uh run the latest kk code in a distribution and that's going to require uh changes to the release channels to actually be able to run those tests with a build. That's already had that a build of azure vsphere.

C

You know uh gcp aws, that had with the latest, and I do mean latest version of the kk code.

C

Did that answer the question nick.

A

I think so yeah.

A

So yeah I'm just trying to take some notes here on what we need to do. So I get that we need to go to six with failing tests and and kind of ask how they want to proceed with those. um But in the beginning you mentioned sig architecture and sig release, I think, and and did you have like specific questions that need to be resolved talking to them or.

C

Well, I think anything that crosses sig boundaries, uh we're basically going to need sig architecture involved, and I think, if we're talking about needing to make changes to the release channel, to be able to do things like uh provide kkk, you know uh pre, like pre-release pre-release, um kk images that can be consumed by uh the distros and then uh mechanisms to then reabsorb test results.

C

uh We're probably going to need sig release's help.

B

Actually, that's raising interesting questions. I have a question when you say um latest and like we're talking about a point in time like the actual point release, or are you talking about pre-release.

A

C

B

You say immediate, do you mean continuous? I feel like I.

C

C

Or varying definitions of continuous right I mean if it's once every four hours, I'm fine with that it, but yeah I mean there. It is some definition of continuous.

B

Yeah, I almost feel like we want a view and test grid that tells us at any given moment where we're at with this, then, is that fair to say if we don't have it that already.

C

Yeah, I think I think that is very fair. To say I mean we got to remember wherever we end up we're a step down from where we are today where we are today with these tests is that they are uh pre-push right.

C

So if I make it, if I make a pr and I submit it, um it has, it hasn't been. All these tests have been run before it even gets merged into the code base. Now we're kind of saying. Well, we probably can't do that so it'll probably merge, but at least we'd like some signal uh that it broke something in in a reasonable time frame, even if that's just within a day right.

C

So that's already a fairly substantial step down from where we are today, and you know, if it's not continuous, then I'm I'm trying to imagine what it would be like if, at the end of um a release, we discover all the things that have been broken on the cloud providers in the last three or four months, and that that sounds pretty horrific.

B

Yeah, that's terrifying!

B

Let's not do that. Let's do what you're saying instead.

A

Okay, yeah, I don't know how to summarize that, but basically we want, as as good as we can get feedback um test feedback test results, whatever we're going to call it um from distros.

A

uh Let's say test results from distros in place of the pre-push testing that happens today, which is better in the sense that we get. You know like clear. uh We know when we break something um and we need to take our plan to sig architecture and sig release to to kind of solve that problem.

A

All right um do we have a timeline on that, like we, we we're pretty close to getting this list of failing tests, so I guess we'll be kind of I guess talking about this next meeting.

C

Wherever we are talking about it next meeting and hope, once I have that list, I was hoping to put together sort of a rough draft of the email. So I'm happy at that point to share it with, with the seg cool.

B

All right all right.

A

A

We have a couple items on. The agenda, looks like bridget owns both of them so go ahead. Bridgette.

B

Oh yeah, I just added a couple based on the things we're actually talking about when we went through the provider updates. um If people are wondering what link I was using to get to all of the sorry cat, uh what I was using to get to um the list of tests that we were in the process of fixing, it was that great issue that nick had opened.

B

um I think that, especially if you were only looking at um the things that needed triage, maybe if you haven't looked at this one lately just go back, take a look. He has links to all of the awesome uh lists that you can go and find out what your current failing tests are, and we had already worked through a couple of them and I think we should well. We have you know a moment work on some more of them yeah. It's.

A

B

Pretty easy to just you know, search for your provider in there and be like hey, look at that or jq it out. However, you prefer yeah.

A

Cool does anyone want to see? Let's see, let's see, let's look at google cp gce.

D

B

Just kind of see which ones apply to you if you look for azure there's like five of them, you know.

A

Azure yeah cool yeah.

B

We have a quick, quick, quick cheat sheet of what to go. Look at now.

A

Awesome thanks for that link um and then cherry picked up mine this week.

B

I actually just went over on the release management channel on kubernetes to remind the release team that hey we needed to get a couple of our cherry picks in. I think it's one of those things where, if I forget to go and remind people to review our cherry picks and get them in, sometimes they don't get in and then you have to wait another whole cycle and every once in a while. It ends up being.

B

Oh, I have cherry picks for something that literally isn't getting cherry picks anymore, because it took that long, and so I feel, like that's kind of a from our point of view, for anything that we're trying to get into the entry reminder to go, look and see if you have any that you need to get in because now is your moment if you would like to get them in this time,.

A

That's a really good reminder.

B

I don't know if anyone else needs that info, but I needed it so I figured I would give it to everyone.

A

Yeah might as well yeah. I think it's worth just looking to see if you have a anything that needs to be cherry-picked.

B

Oh yeah, especially one of the things I notice sometimes is you know how, when you're reviewing all the things that you've been getting in lately and then somebody is like. Oh yeah, we should cherry pick that we should is not actionable or clickable. So go make sure you actually cherry picked the things you intended because yeah.

A

Otherwise, it's super easy to just forget to do that.

B

Cool awesome and when we're done with extraction, we'll have less of this to worry about. But you know.

A

Yeah seriously all right. Well, I think that wraps up the agenda for today um does anybody have anything else they'd like to quickly chat about before we drop.

A

Cool all right thanks, everyone.

B