Kubernetes SIG Node, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Node CI 20220921

Description

SIG Node CI weekly meeting. Agenda and notes: https://docs.google.com/document/d/1fb-ugvgdSVIkkuJ388_nhp2pBTy_4HEVg5848Xy7n5U/edit#heading=h.2v8vzknys4nk

GMT20220921-170732_Recording_2560x1346

A

Yeah, okay, uh welcome everybody. It's a signal, CI meeting, we uh it's uh September 21st 2022. We started a little bit later due to technical difficulties, but almost on time welcome everybody and to the agenda. I I want to continue doing our triage. We didn't show the Box first time and we have quite some uh like quite a pile of everything uh uh accumulated. So we need to keep doing that and I also wanted to discuss some dashboards. uh So when I came back, dashboards doesn't look the same as I.

A

Remember them so I just wanted to discuss, like maybe somebody has some um uh information on. What's happened and maybe you can I I think we need to reset this uh renaming issue I reopened it, and maybe we can do that. So, let's maybe start with this, and then we can go to triage so our first of all, so the series block and it used to be not conformance and uh um some Kubler tests.

A

But now it's uh like 122 123 group test two like test and it doesn't feel like I mean first of all, the testing, the version of I mean if it's version of kubernetes, which I assume it is. uh We are not testing the right one because 122 is not uh is out of support and open source right now, so you may want to extend it to 124 125 I'm, not sure why it's even here but I guess it was here all the time.

A

um So this is the first thing and I remember. It was a not conformance test here as well and now is a meeting. So maybe this test by itself has not conformance uh yeah, but I mean it doesn't call not conformance so it. uh It may be hard to understand.

A

So this one. This is the first note I had. um Does anybody know any anything about this page?

A

No, okay, okay and this page is very small. Now so I again, I remember like Alpha test, but Alpha test we removed because it was no Alpha and then test so maybe once we will get them back we'll have it uh and then features again test on 122 and 123.

A

I'm, not sure why uh this is clearly, some kind of degradation happened here and then, uh like both swap jobs are failing and I think they're fighting for a long time now.

A

uh And again, there is no like features or like uh what are the other names mom, so I remember way more uh tests and like at least test on my steps were here. I think what happened is many tests got migrated to this tab and this tab, like specifically, if we continue runtime, so this might have happened. I'm not sure, like I I mean tracking history of this tabs as hard.

A

um So this may happen because I see features here like you see, I, see features here and it's uh um no say I've been uh been run uh in some tabs.

A

We have serial. We have features.

A

Unlabeled so old orphaned, so it feels it. There is something happening. So maybe what happened is oh many tests from kublet got migrated to continuity and cryo, which makes it clearer when you want to understand what environment you're running on, but it may not be very, uh very clear, From perspective of person just coming to this test grid and trying to understand what they appear like what the status of couplet like.

A

Is it green or not, and uh yeah and minus saying that this tab is failing for a very long time now, like I, see it here signal? Of course it was failing since uh 7, 18.

A

and I cannot even track the history of being here, because it doesn't show that long in the past.

A

Yeah, so we and everything like this smaller tabs seems to be quite happy and healthy, um yeah.

B

Probably detector or screen ish.

A

Anyway, so the smaller ones- huh small one since uh reasonably okay, anyway um yeah does anybody have any history, like has anything happened uh intentional during the summer or is there any intentional changes that was made.

A

If not I I I suggest we I mean I. I will start the like moving things around a little bit. I will use this task to discuss. So please see yourself on this task. If you want to participate in this renaming, um my main idea is I want to clean up release block into I mean it's a first thing.

A

People will look at so we need to understand who is in control here and like what tests are being presented here and I want to migrate some of the container de tests to kublet, so it will be like when you look at Kubota. You have at least some understanding what's going on like.

A

uh Is it healthy or not, and then specifics will be on container D and cryo um and then like I'm, not sure whether there is a bug for this failure like this is completely red, but if it's not I will I'll cut one and see what can what you can do.

A

um I remember having problems and saying like. What's this test like why who is owning this tab, it feels like there was some other owners Beyond signal, but now it's on under signal, so uh I need to double check with costume if they want to own it or like what specifics of the stuff uh we need to preserve, and if we shouldn't we just, we can just remove it. This is the main idea, any suggestions.

A

Okay, yeah just see yourself on this book and uh I will I will write it down uh later.

A

Okay, then I want to go into triage, I removed all the items from here that are not relevant to CI. So if we don't waste time here, uh the only reported bug is here. I think uh David was uh filling up. This uh um see this one this chart and uh he filed the new bug for device plugin.

B

C

I think Francesco and I are specied on that one, so we'd be taking a look at that.

A

A

Do you want to get assigned to that or uh yeah feel.

C

Free to assign it to me.

A

B

Very interesting.

A

Yeah, that's uh pretty much. All we still have a lot of issues to do um is any like Muslim are assigned. Oh this bug. This bug was cut for this tab. That is failing from since 718.

A

Okay, I might not be able to show the slack.

D

A

um Does anybody wants to own this and uh see through the end or otherwise I will take it.

E

I think I can take. This I have some context with costs.

A

Okay, so uh what is your.

E

um It's a Miranda links.

B

B

A

Yeah everything else is: uh has a shiny, oh no, okay,.

A

A

Probably disappeared.

A

So this one was performance tab that is still failing.

B

Is it still yeah.

A

Still fighting.

A

Yeah I remember I fixed one issue, but another issue appeared um and yeah. Some need to take a look further anyway. um Pseudo cry jump railing.

A

No not failing anymore.

A

A

We cleaned up I think we just removed the whole job.

A

Keep it on restartable parts.

B

A

It's just another uh big Russian issue: I think we can I I, just just keep it for now.

A

Yeah just uh same.

B

Okay, I think this was.

A

Yeah um I think something my TSU wanted to take. This uh was rpr.

A

Oh yeah, there's already PR.

F

F

Yeah I didn't test mine, so yeah I can I can check Ben's One.

A

Now I think Ben is for something else for a kind.

F

Okay, yeah, okay, so yeah I will continue on this one. Yes, okay,.

B

A

Did I open it? Okay, it's assigned to you cool! Thank you.

A

Okay, so I mean at least we reviewed it. uh Someone suggesting uh like fighting test issues, I, don't want to assign to anybody but and uh few small issues that anybody can take so yeah if you're interested. Please take them.

A

Anyway, um I also wanted to show performance stuff that we used to look at um and uh yeah. So performance is the one I think I want to show. Is uh you see something happened on?

A

Maybe we could go that memory of 90s percentile of CPU got down and memory the same way, which is a good thing, uh something uh improved incubat I. Don't think you need to investigate, but uh I just wanted to show that something happened. Something good happened, I like when something goes down and runtime doesn't change. So it's clearly some changes in couplet that helped with that um yeah. So if anybody wants to look at the PRS around this time, like this is time stamp I like this Linux time, 156 whatever.

A

uh So, if somebody wants to look at it, uh be my guest but uh I just want to show that nothing bad happened and everything just get better.

F

I think it's not the change in in golang, maybe not at this time.

A

uh It's hard to say: I think yeah, it doesn't I mean it's not unreadable. The data unreadable so I think it's uh my I, don't think it's one week, but I I may have like very bad memory around this.

F

Now so it should not be it. Okay,.

A

A

A

Nothing else I will switch to box Dash. Is there any other testulate agenda.

A

Okay, so now switch to box triage, and uh this is a separate tab.

A

Last time we did a good job of uh three original bugs and now we only have 13 left and then maybe to use some news and information once.

A

Okay, very fresh five hours ago.

A

A

A

Need container combination.

A

I, don't think it's six storage signal. I I would like six Windows, uh take a look first.

D

A

A

A

B

B

B

A

um I'm not sure how much it is.

A

How much it is uh signal issue this issue.

B

B

On the storage.

A

Because we need to uh it feels that.

D

B

This back on, second to the corner,.

D

B

A

So this person, like I, think this is uh that's how the problem so it starts deleting slowly, but then it somehow expedite Edition.

A

um I know I would I would say it's six storage more than six node, uh just to understand the right sequencing that we need, but we still may want to look at this box mentioned here.

A

um I would say, we need to triage, accept it, it seems to be reproducible.

A

A

Interested in this call Please assign yourself, but otherwise my music storage can take a look first.

B

A

All ports, okay, seven days ago, um exactly last meeting.

D

B

Okay, shut down gracefully.

G

This is a working as designed I think, once a pod gets assigned to the node, it won't change unless it's part of like a scheduled job or some other type of workload.

A

But it should receive this failed signal right, so it should get into failed state.

G

uh It depends um if the.

G

It depends on its shutdown. um You can assign this one to me. Searching I'll, take a look at it.

A

Okay, so she Ming actually looked at it. Is it 122.

B

A

B

G

R Phillips.

B

B

That are south of the future.

B

B

B

A

And let's continue D16.

A

Failed fail to create container task assumption.

B

A

Okay, do we have a subtest foreign.

A

A

A

What are they all fighting.

A

Yeah but one of those tests will create a board.

D

D

A

I'm not sure um what information you can ask if we need to understand how to what this message May mean.

A

F

Stupid question, but is it related to the C group version.

A

A

uh Mike, are you gonna call, do you remember if we tested the swap on C group, E1 or C group E2 as well.

E

um Yeah I took I worked on this a while.

A

um We didn't change the secret versions, yeah.

E

I think we still worked on three groups. We won. We never made the upgrade uh because we.

D

E

Wake up the development on Twitter.

F

Because I I wonder if, if swap, is enabled on secret V2, it wasn't initially I think.

E

I'm not sure if the workload is uh like if everything is fully implemented with seriously too.

B

A

Yeah I I accepted it and see what secret version they will see, but then we need to take a look.

A

Let's say priority.

B

Get my feeling better. So that's the complete component.

D

A

Yeah Docker sock.

A

D

B

Person, yeah I, think it's the same person.

B

A

A

It's formulated as a question.

A

C

Look into it but I think at the first glance, I think we can't really do that. That's my initial hunch! If it's directly appalled, maybe if it's a deployment you can edit the resources. I'll assign that to me.

H

B

H

Before that is that a support question which we should close or an actual bug, we should an issue I mean we should work into.

C

D

A

It's not a feature, no.

C

It's not a feature, I think it's just a question they're trying to figure out if they can resize um the resource requested by the part when it's subversible quality of service. If I understand this correctly and they're, not explicitly asking for any feature.

H

Or fix yeah right anyway, I just wanted to to raise the question. I don't have an answer myself, yeah exactly.

C

B

Thank you for watching so.

A

Can it be documentation, issue I, still don't understand what this person wants.

A

Cpu exclusiveness and the other containers use the shared pool.

A

A

Okay, is it support or documentation.

C

Not sure I need to look at it.

B

Thank you no worries.

A

Oh yeah I I, wanted to reply. I still haven't.

B

And its features.

A

Okay, pause return. This and I can see the data from eccg.

A

A

A

A

H

Was trying to say that my gut feeling is this is a legitimate request. I'm, not sure if we can fix spare API contact, I literally, don't know, but it seems a fair enough issue: I mean if the equivalent is still recovering the it should signal to the its own endpoint, okay, I'm still recovering, and that seems a legitimate task and reasonable proposal to fix it. So it seems we should probably accept this and see. If and now we can improve this.

A

Yeah but my my question would be either ever I mean consistency. We need to understand what consistency means is it uh like? Some parts can be scheduled and still haven't, showed up, um I guess when they got it from API servers and Google tool, no will return them.

H

That's true as well I mean yeah. Send that to me, I will have a look and I'm interested in this area. I mean for just thinking about if the API contract of the pots and points say this can change. Okay, but I need to check so yeah. Send it to me. I will have a look I'm happy to learn.

H

A

A

Let's give it as a bug.

D

A

Okay, I think uh we just casted uh no uh internally and David wanted to take a look. Let me accept it.

A

Just about 122.

A

A

Port and one port being terminated and endpoints not always been removed.

B

Something like that.

A

Support to the low image class prone is equal to understand.

A

Then you'll assigned to himself and.

A

A

A

B

A

Consistency not controller, make it safe to accept. If somebody wants to take a look at uh not controller, please take it.

B

Okay, um how to look.

B

I didn't move this one.

A

Because it's sick Windows now.

B

And this is support.

A

Okay, uh we only have seven minutes left um if there is any comments on this meeting or anything, uh please speak up. Otherwise, if nobody has any agenda, I will go to need information and review latest.

A

Okay, I think we're getting much better. Like last time. We didn't even went through all the triage issues now very caught up and triaged uh we'll catch up on need information. Hopefully, next time it will be all new and hunted away.

A

Okay, some dates.

A

A

A

Okay, so this is uh a lot of volume of these messages.

A

Ryan you've been looking into that right.

G

A

So there is two aspects of it. First aspect is why those uh things happen in systemd at all, uh I think many customers reporting that my second question was: how does it affect Google it so people claiming that the equivalent consuming Google itself without logs consuming so much memory that the two ohms when this message happens, did you look at that? Like did you know, okay,.

G

A

G

I have not looked at this latest comment, yet.

A

Okay, so because, like I, don't know um if somebody's looking into the system G side of it, uh this separate but uh My worry, is that uh kubert memory usage grows when this issue happens and I. It does sounds good when Google crashes, when there is some environmental problem.

G

Yeah yeah I agree.

A

Do you know which uh which retry is happening like uh wearing code? That's happening on cool blood.

G

I'm not really sure um I have to look into it. Some more the it looks like they might be, using openshift as well so um I'll see if I can debug it a little bit.

B

A

On this back, do you want me to CC you yeah.

G

Please foreign.

B

B

A

A

Let's wait another couple days because uh it seems that uh was actually seven days 20 days ago.

A

B

A

A

Maybe or digital ocean problem.

A

A

I suggest we uh and this meeting here and uh maybe next time we will be fully caught up on all the needed information in the garage box and we'll spend more time on uh tests, part of it.

A

Okay Francesca said he will be back uh and meeting could be, and oh, okay, you're back bye. Everybody.

F

Thank you thanks. Thanks.