Kubernetes SIG Node, 2 Aug 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Node CI 20230802

Description

SIG Node CI weekly meeting. Agenda and notes: https://docs.google.com/document/d/1fb-ugvgdSVIkkuJ388_nhp2pBTy_4HEVg5848Xy7n5U/edit#heading=h.2v8vzknys4nk

GMT20230802-170531_Recording_2304x1440.mp4

A

Well, hello: um this is uh signal CI week. Limiting uh today is uh August, 2nd 2023 welcome everybody. Today um this section of this meeting will be led by dixita Dixie. Do you want to take over.

A

And as you see that we don't hear you are, you are muted.

A

We can hear it now: okay,.

B

I'm sorry, this is my first time, so it's a little glitchy for me, okay, uh so this is August 2 2023 and we are host I'm hosting this week's uh Sig note CI meeting uh I'm going to try to present my screen. First.

B

It's is my screen visible.

B

Okay, so I'll be doing the test triaging today and uh Todd will be doing the bug, bug triaging uh the we have the agenda here. uh So we are looking for more people to engage in the meeting by Leading, either bug triage or test failures, and if you are interested, please reach out to a circuit can start with the test failures. uh The GCE device plug-in pre-submit job is still broken. So does anyone have any insights into this.

C

Yeah, um hello, so context for this thing, which I added is that now I was validating or trying to validating a back part of mine and I literally, don't know if this test is still maintained or supposed to be working literally so I how we found it using. You know, test question mark, so it could be just stale and I don't know. So if anyone has any insight about uh or just say yes, this is supposed to work. So something is up just reach out and to me please, and we can move forward.

C

The failure I see seems to be when the test is setting up even before the actual you know, the actual Recreation starts so and I also filed a very simple PR, which just you know, against the same branch which just changes a comment and run this and test and again same issue which points to obsolete test or broken test or whatever. But I really don't know if all things are supposed to go. So that's it thanks.

B

So, where is your PR.

C

uh Yeah, just a second I learned to agenda the the testing one.

B

Okay, okay, all right! uh Moving on to next one, the multinomial test is still failing on this particular PR has some of the details and I. Think swathi has created this VR to modify the machine type to narrow down the failures. So what you do you want to talk about? It.

D

Yep, um so um the issue per se captures that periodic jobs are failing, but um I was trying to figure out if, if it is just periodic jobs or it's um all the jobs that essentially use this image, config file, so I created a dummy PR to test out um test the route and I I identify that pre-subish jobs that are referring to this config file are also failing, so essentially, The Next Step was to change the submission config file, which I've done as part of this PR, so that should narrow down.

D

If it's to do with the image config file itself, and then we can figure out. How do we enable multinum systems or machines in test infra, so I think the next steps are. If someone can take a look at this particular PR and approve it, and then we can see if CI require, if CI responds to this in a positive way. And then we know that it's this issue in and following that, we can further make changes to the image config file to enable multinomial systems in CI.

B

So Sergey you are already on this- should.

A

I I I just uh uploaded. Thank you, sorry for not touching it earlier.

D

No problem, thank you.

B

Okay, so the next one is the arm failures that we saw last week. Those seem to be fixed now on the pr with the fix is already merged, so we are good there and then moving on to the containerdy jobs that were failing last week, they are still failing, I think yeah. There were two test failures, as mentioned as mentioned on this PR one was for um test, the fix is already yeah, I, think they're waiting for 120 uh nine to open, and then this will be merged, but the summary API tests.

B

This test is still failing, as we can see here, and this hasn't been picked yet I think am I right, Sergey yeah.

A

We cannot merge it before 129 I mean we can, but I, don't think it's critical enough to justify as a merge right now um so to remind everybody. The problem was that uh specifically in Canada energy and I'm, not sure about I, think a crowd was the same behavior.

A

um If um happens too early in a life cycle of a container, then it may die with a different message than um uh status. So test is looking specifically for home status and if it happens too early, then you cannot find it. That's why we just I didn't sleep into before we Asian sleep before allocating too much memory, and that fixes the test. Fakeness.

B

But we'll just also this wouldn't fix this issue right. There are two tests that are failing, I, believe.

A

Yeah, this wouldn't be fixed by zpr, yes, yeah.

B

So I think only Home Care.

B

Still looking into so Paco has mentioned riage accepted, so does that mean that he is going to look into the second test? Failure as well.

A

No I, don't think so, um like you need to assign uh to himself. If you will look into that.

B

Okay, so do we want to? Is anyone interested in taking a look at this one.

B

Okay, uh if not, maybe we can take a look at it later then.

A

B

uh I think that is all with respect to the test failures from last week. um How do we want to proceed ahead? Do we want to go through the rest of the agenda? First Circle. Do you want to go through the rest of the agenda first or do we want to start looking at the test board.

A

I usually going to go to agenda first uh to get it out of the way. Okay,.

B

The next item is Untold. Do you want to speak about it.

E

uh Yeah, so this was one uh I was looking at from last week um and uh it's on arm 64., there's a test that runs bin top and for some reason, arm64 using the five-year-old busy box image.

E

It works fine on 1964, but on a harm. That's so false and the stack Trace is a little useless. There's it's a strip binary, there's nothing there that I can see and upgrading to a newer version of BusyBox fixes it. um So this is a change to upgrade to a newer version of BusyBox and that should resolve this arm.

E

64 flake arm64 test failure anyway, and then there was sort of the second part of this question was there are some other test images which are really old and one of which is um depending on Centos 7, which is now like end of life, and there will there will not be a Centos or a newer new updates to send to us. So I'm curious like what's the like.

E

How do test images get updated is do we just leave them alone until there's a problem and then change them and and sort of what to do about the Centaur seven like there's like Alma Linux, which I did build this with all of it. It worked. Fine still uses the same RPMs RPM names, and it is up to date, but I'm just curious what the consensus is for that.

A

I'm trying to be on supported versions, but uh so protest images, specifically, we tried like since they don't um I, will try to replicate as much as possible users uh scenarios so users, typically on supported versions of images. But beyond that it doesn't matter the image they run. It uh I mean uh hosting platform matters more because typically kubernetes will be running from, uh especially in managed environments. Kubernetes will be running on more modern uh OS images, and uh so we need to update it more often but test images.

A

We don't update it that much okay.

B

So does that mean we wait until something is broken and then we update the image so again.

A

Either something is broken uh or we just keep it above. The end of life.

B

B

Okay agenda and we do the dashboard.

B

I think this is work in progress.

B

Yeah, this is work in progress, so can I move it to be.

C

Honest this one uh I, don't know how we want yes, this is from me, so basically, this one is going to be just deleted when I figure out, uh what's going on so I'm, not sure how we want to map on the board. This should basically no one should look at it unless to to understand CI, so not totally not worth review or unless you know for targeted.

C

You know for understand. What's going on so yeah I, don't know how to express that in the board.

B

Can I move it to waiting on author.

C

B

Okay sounds good.

B

Next one is ADD sample device plugin to the config.

B

This is also by you. Yes,.

C

Yeah this one in first one should have been I, think it has the flags and it's stuck because we are in test freezer. So I, remember we. It was no controversion. uh What.

C

Or or anything I don't know well, okay, anyway, in in general, is ready, it's simple. So if people want to have a look, yeah I, don't have looks with me, so maybe what's remembering it's reviewer so feel free should be very simple.

B

Does anyone want to review this yeah sign.

D

This to me, I I, don't have any contacts on this I can't remember. Actually what this was about.

A

I'm Miss Jessica hero, so I vaguely remember sample plugin configuration.

C

Yeah, basically it's it's a it's a cleanup, simple device. Plugin is not part of the image list which we use to.

D

D

C

B

Okay, this is for um node conformance test container PID namespace sharing container in Parts using isolated, PID namespaces should all receive prd1 okay. So what is this? Is this.

E

Yeah, this is the the top seg faults on arm. 64 use an old busy box.

B

Okay, the main issue for that.

E

uh Yeah, yes, as well I started looking into it. um There's.

F

E

Out but you need to wait until one two: nine okay.

B

E

B

So does it stay in triage or in progress?

B

I think I would move it to in progress. Then.

A

Yes, please, and it's already assigned so yeah.

B

I think this is the same one, um the one that swathi created, so this also has been assigned so I'll move it to in progress.

A

So this is a PR, so we typically just put a neat approver and.

B

A

We're just basically merged.

B

Okay, so I'll leave it, as is.

A

B

Needs a problem: okay, my bad.

B

The next one is use C to P2 instead of E1, but this is still in draft.

B

Thanks for directing me to the right place, I'll make this build. Okay, I think it requires something from the author because they are yet to make some changes to the pair. So I'll move it to waiting on author, then.

B

Okay still have 10 more issues.

B

So this one is Skip allocation for running pods, 125 yeah.

C

This is um product back port, so not for the like includes test, but the main focus is not test.

B

B

So where which dashboard does it go to then.

C

I I, okay, psychiatrics, keep me honest: I think it belongs in the node peer charge, but not in the CI test board.

A

Yeah uh I would uh just move to Archive it um because otherwise it will keep popping up and thank you.

C

B

uh Move away from get pod, Qs use port status, Qs classes instead.

A

State it's a product change, so we need to Archive it out.

B

Promote cap to Beta and 129 so again, this.

F

B

A product change, I believe yeah.

B

B

E2 nodes support CPU for more than three threads and one core in CPU manager, test.

C

This I believe should be the other way around, so not in the North board, but totally in the test board, because it's a test, change and I will actually review it. So Google does assign it to me if you want, or anyone else want to join, feel free.

D

B

D

Well, on that, okay.

B

Let me do that. First,.

B

You said CC swathi.

D

Yeah CCS sign whatever don't mind.

C

B

And what does it still stay here in the node priage? Oh my.

A

God, it doesn't matter because even if you'll remove it, it will come back because it has all the labels. So I would just care about this. One yeah just move it to an easy viewer.

B

Okay, add additional utilities to cube cuddle image.

B

um Is this for the test.

B

This looks like a feature, then right.

A

Yeah I think so.

B

So can I archive it from here.

A

It sounds good.

B

Use image config to remount, temp I think this is for the test. The job uses cos.

B

When ramp is mounted with no exact option in the class, which makes it impossible to runco and other.

A

Yeah I think it's about image.

A

Okay, I think it's assigned to me already right.

B

A

Keep it this way. Thank you.

B

A

So apparently um like when we create an image, we need to make um uh one of the like TMP executable, because we put test binaries there and we don't do it unless there is an image config file.

B

Okay, so what is this changing, then?.

A

uh Yeah, they um I think they add, will I haven't looked very deeply, but I think ad agent's image config file to use for this image for the specific test. Okay, I think the question was about like whether we want.

G

I have a question on this um yeah. So why can't we add those parameters straight into the cloud init file instead of making a new image config.

G

But if you look at files changed, um there's a new image, config being made.

G

Instead of just adding those extra commands to the cloud init file that already runs in our job.

A

uh Which Cloud menu do you refer to? Because I think this is how we had to uh to the image to the cloud in.

G

Oh sorry, wait a second like it's good to fast change for a second.

A

Can you go to file change, please.

B

A

G

Okay, oh I, see um so you are a missing image. Config file, so doesn't the the GCE Runner automatically pick a default Cloud image file? If you don't specify.

A

I, don't remember any defaults, but.

F

I, don't think, there's any default on this. It will just not run anything, uh but there is a way to specify like a script. If you wanna foreign.

A

And the question was um I glanced over yesterday, so the question was whether we want for Canary to use something more uh Advanced like some like GF, build of um uh of the same image, or we still always just want to use uh whatever like I, don't see him like. The font is too small, yeah um yeah.

F

Right now we're using one uh class 97, which is um it's a relatively old image,.

A

Yeah, basically, you need to understand why this test was called Canary before. Is it because of image, or because of kubernetes.

A

Yeah I will I will look deeper into that understand. The history, but I might have questions too.

G

Just to add to this um Cube Test 2 has its own set of no D3 tests that are running on there. So if there are changes that repo you will know, the the existing node tests have to pass hope. You guys get emerged.

G

Foreign dashboards.

A

Can you say it.

G

Again uh so some of the sick testing canaries are used to test keep tests to um when it's been developed.

A

E

A

Think we have many tests, duplication like group test two and non-cups too, and it's not only canaries.

G

A

Is it time to clean up uh like remove non-group test to tests.

G

I'm going to investigate that and plan that work for 129. um sounds good. Thank.

A

G

A good time because we're at the end of the release so I don't think anyone is going to be too happy about me breaking up all the tests. This late in the release.

A

Exactly thank you.

B

Okay, so I'll keep it uh in needs reviewer, so that Sergey can take a look at it. Next one is fix freaky text test on DRA test, prepare resources should time up. Oh, this is for the bug fix this fixes some flake in manager test.

B

So I think this requires the reviewer to sorry the author to get back on these questions.

A

Yeah wait for author.

B

B

The next one is uh for a pre-submit job, e2e Gap, C windows,.

B

It seems the job seems to fail during the setup Francisco. Do you have more context here.

B

I think this is an issue that was open five hours ago, so this test is failing. This.

C

Is the other test that it seems to be failing and which I found while validating my backboards and again I? Don't see any obvious reason for my backboard to make this test fail and it seems again fail in the in a related place so and find the issue for tracking. So that's that's the story actually.

C

So if anyone has Insight or can point me to people which can help be my guest, otherwise I will try to investigate and I think just one last note I think this one and the other issue if I should not block the backward, but let's see what reviewers think about it for reason. Okay,.

B

So this does require still required triage. Then right.

C

Yeah, because um I think the basic question is okay: is this line still supported? You should supposed to work or I just hit, something that is stale and should actually be remote. I think that's the triage. We need I. Just don't know if this Lane supposed to work.

A

Yeah I I think more and more we need to have fake PR from like every week to run all the pre-submits.

B

Okay, so is anyone interested in talking about.

A

B

Oh, is this the same device plug-in GPU pre-submit seems to be consistently failing. I think we discussed about this.

C

Yes, exactly yes,.

B

So should I move this tutorial as well.

C

Yeah I think so.

B

Okay sounds good, so I think we are done with the test triage here uh we can move to bug, triage I can stop presenting or if you want to take call.

H

B

Okay, I do not know how to stop presenting I. Think I did stop presenting.

E

Is mine displaying or yeah okay, human Zoom, all right, so I went ahead and opened up the uh all the uh bugs here in uh triage list.

E

So let's start at the top, this one um I was just looking at it. um Let's submit a job that requests the GPU. It.

H

E

So the pod's incompleted or the job's in complete status and I, said when node restarts the job status goes to error.

E

If, basically, if the resources were then used on the system, I thought.

C

E

Seen something similar to this before with Cuba restart and no.

C

Wait wait, I, think I know what's going on there, we, uh the thing is Cube letter start. We we have an issue which we fixed, which does exactly triggers exactly this error, and that was done. Oh no, yeah, you're right. This is too old, you're right, you're, right. Sorry,.

E

Well, I mean it was fixed recently yeah they're running 123., so no.

C

No, we didn't back for that far so it's similar, but it's not the thing. Sorry for the noise.

E

um It's been asked: if they can. Oh there, it is yeah, um so they're saying I can't reproduce it one. Two seven and.

E

H

E

E

Looks like it might be, a real issue was the Pod completed when it restarted.

E

Oh, the John pods updated plot size from completed to the unexpected machinery.

C

I do remember a bug a while ago, which I should probably not look at the deeply enough, which was exactly about uh uh completed body if you restart the cubelete territories, admission yeah and that could be related this time. This time could be button. Unfortunately, remember, look, look what we can do is that, could you see me that I think I can I can have a look and try to reconculate things thanks.

A

Do you want to ask Google logs and uh what whatever else may be helpful, like Nvidia driver version, for instance,.

E

And then I accept one except this I could spell yeah.

A

It feels that it has enough details, so maybe look will be helpful, but I think it's already.

E

But this one as well see this.

H

H

E

This might just seemed like a docs thing.

E

They're basically saying if you have a mutable thick map mounted, then you delete it and recreate it of the the updated one doesn't get doesn't appear in the Pod I. Guess that's how I'm interpreting this.

A

The immutable they are, but um they can still be mutated by being deleted right so immutable. If you misleading here.

E

Yeah I guess you're saying: if you delete it and recreate a new one, then the uh um cuboid doesn't notice. There's never watch on it. So.

A

Yeah I wonder if it's a feature request saying that, uh since it's immutable, then kubert assumes it will okay forever.

E

A

E

It seems yeah I would expect if you have an immutable config map and it gets deleted, like you wouldn't want.

H

E

Disappear, that's sort of the so so maybe.

A

E

Yeah I think that, and that's always asking like, can we just clear up documentation, um um yeah dark Bishops like that immunologics need to be delayed and recreated with a different name in order for the new pods to see the new version.

A

Yeah, you can uh make a kind documentation.

E

A

Remove kind yeah, I think Bach.

A

Sf team I'm. Sorry uh there is a person SF team at SF team.

E

Yeah and accept triage accepted yep. Thank you.

H

E

C

Right in this case, the other, uh the author of the previous issue, acknowledged that their PR was a duplicated of the pr fixing this issue, so I think we can get. We can close the yeah I.

H

C

We can cross this one.

E

Mark, that measure is closed, uh pod created without you know, the economy removed.

C

I commented also this one um so but I'm not so I I think I. Remember that okay, creating pods or containers using cry CTL is just not supported, but I don't have a quartable source to back my claims. So this is why I left it pending.

E

Yeah I mean I, agree: you're, like you're directly, creating pods in the runtime I mean I. Guess are they even on the.

A

uh Has any more important as well.

E

Create the container so.

C

Yeah I think in this case they are just trying to create codes, but either I actually bought sorry or not are not supported. I think.

A

Yeah just curious if it needs to be quite CTL issue instead of Google.

E

F

E

Might be maybe they're just trying to use crescent deal, not even cubot's, not even involved at all they're. Just saying, if you create a pod with no uid, then they can't remove it.

C

A fair point could be: yeah could be crazy, Theory issue yeah, but then on the other end, they also quote cubelet source code, which is what uh point mean that direction yeah.

A

I just see right, I think yeah.

C

A

We can try to mention Sasha to see uh what he's thinking about it.

C

um So, what's the.

E

C

No, no only keid is uh is exactly yes,.

A

No, no I meant um Sasha growing. It.

C

Sorry. Sorry, sorry, sorry, but.

A

C

True, true, sorry, sorry yeah.

E

uh Sorry, I don't know that that get up handle either. You.

A

You started right.

C

A

One, the third one, yeah okay, you can ask: what do you think about? uh Is it a series.

A

Can you see me as well just uh yeah I'm, really curious.

A

Yeah and uh on the issue, can you move to uh need information on top of it different section yeah? Thank you.

E

um Why does POD start container logs cathedralize.

A

Yeah kind support seems more appropriate yeah.

E

A

E

Remove from the yeah.

E

How do you remove just Mark, it's done.

A

Right typically, I click on uh dashboard name, and then it brings me to the dartboard and then I remove it from there benefit of all response. That's why I have archive with a button.

E

Thank you all right, um pretty much part I think we talked about this one. The other day.

E

That was only four.

H

E

That's not enough.

A

I remember seeing Hicks for something similar.

E

uh Yeah, it sounds familiar.

E

Yeah I think it's required. Some digging.

H

E

Not not exactly sure, let's.

C

See I think it sounds familiar because it was highlighted in the channel, possibly and every. But besides that it sounds familiar to me for some reason, but I didn't have time to look into it.

E

And in privilege.

H

Mode remove the.

E

Secret view to entry for CPUs.

E

H

Seems like there's.

E

E

Seems like there's enough to try to reproduce it anyway,.

A

uh Yeah, the question would be: is it what are the conditions to hit it, because we test Europe too extensively.

E

H

Saying just run again.

E

With the secrets we do.

A

So running previous issue, I would be puzzled because I'm pretty sure we run a lot of pretty good spots. I.

H

Would be surprised if the privilege, if the test that we're running specifically checked to see if they can access CBS, that their CPUs from within the container that's kind of a narrow, uh Behavior.

H

I wonder if this has something to do to both crime and continuity. When it's a not privileged pod, they could see group same space and see us V2, but I wonder if maybe we by doing that, just with the you know, amount to see its sfsc group within the container for privileged containers, because I know we do that on previously. What the mount says, FC, group and um or so maybe.

H

um Do you want to, for this to be in I, can see I'm at hair Commander um see if uh I can reproduce it.

E

H

E

Is wiped from cubelet restarts.

E

I guess it's, what not? The type is immediately removed, probably the status condition goes is cleared and the Tank's removed.

A

So they expect this pressure to stay, be checked on a Google start.

E

Maybe so they're saying there that we're starting cubelet after the eviction is triggered, which will also clear the condition if.

H

Enough was evicted.

A

But is the issue that it's not been cleared or it's what.

E

Yeah, the shine is cleared, the taint is cleared and I'm, assuming that the Tank's clear, because the condition is cleared- and maybe it's maybe it shouldn't be- maybe it should maybe it's initialized to false or whatever and actually hasn't, checked for this pressure. Yet I'm not sure.

H

E

Others related PR.

H

E

Oh, let's look at add some weight for eviction manager to run before reporting status, I'm guessing.

A

Yeah that sounds right.

A

You think about side effects, but uh songs addict.

A

Yeah um is not very tested scenario like we have so many edge cases.

H

E

H

E

Don't see it get restarted very often.

A

Yeah I think we can accept this. So all right.

A

Another question: is it bog yeah I? Guess it's still this question right.

E

Yeah, if the uh yeah, if you have this pressure and you restart it just goes away momentarily and comes back, seems yeah. It seems like a bug. Chip pod schedule that shouldn't for a short period of time.

E

This one I just read the title, which sounded exciting easier: it can easily destroy the entire cluster. It's all not sure it.

H

A

So so the claiming that it's not being deleted.

E

Oh they, uh oh, they create an empty deer uh memory, back temp, FS, basically and then fill it which consumes memory. Then their fog gets boom killed and the temp FS continues to exist. But now the node is dying. I think it's my interpretation of this, because the temp investor still consuming memory.

A

But not deleted by cubot or whatever.

E

Yeah I guess I'm killed and then restarted, but the temp FS is still like their own Guild didn't affect the temp. Fs Mount is still there.

E

E

Yeah so I guess it's a scenario where killing the pod process doesn't actually recover any memory or doesn't recover very much memory.

A

Yeah I'm surprised that we don't do clear um recovery.

A

H

E

Is not correct.

E

Yeah so then it was specified size. You can write up to the amount of applicable memory on the node.

E

um Yeah I'm trying to reproduce this one. It sounds interesting, um yeah.

H

E

Guessing if you wanna.

E

I guess you really want to like unmountain, then remount, the tempo of small volumes upon Pottery start or upon pop, but I mean that seems like it's going to break all kinds of people.

A

Yeah, if you can reproduce it uh and get some cool folks into debate, or maybe you ask Google's logs from this person.

A

I mean they cannot even SSH, doesn't know so, hopefully they can extract the report. Somehow yeah.

E

Anyone else want to be on it.

A

Cool, do we want to make it uh news information or to clear enough.

E

um It seems I mean he supplies a pod that does it so I think okay makes sense. Yeah should be enough.

E

I have to tweak the limits based on my Note, but yeah should be good, so uh I'm gonna try to reproduce this one. This afternoon.

E

E

I thought that was it.

E

H

E

Why these are still showing in the triage.

E

Shouldn't that be over here.

A

Yeah they're not automatically being moved. Oh.

E

E

H

E

H

A

To close it and it won't, it may be cool just just closed.

E

Oh there's special permissions for that. Oh apparently not.

E

All right, so that's all the triage issue um or Treehouse issues. We got I. Think three minutes left so not much time.

E

I'm, just gonna call it here.

A

Yeah I think so, um if anybody who wants to sign up for next one, please sign up on a uh agenda document Let's uh see you next time. Thank you. Dixita and uh I thought I really appreciate you. Okay,.

E

Thanks see you later bye.