Kubernetes SIG Node, 22 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Node CI 20230322

Description

SIG Node CI weekly meeting. Agenda and notes: https://docs.google.com/document/d/1fb-ugvgdSVIkkuJ388_nhp2pBTy_4HEVg5848Xy7n5U/edit#heading=h.2v8vzknys4nk

GMT20230322-170443_Recording_1692x1120.mp4

A

Hello, hello, it's uh March, 22nd uh 2023 signal, CI uh Missing welcome everybody I will share my.

B

A

Because we have, we have some Topic in agenda.

A

Toad, it's your size.

C

Yeah so after the uh to look at these, just because like if you scroll over, they seemed like extremely flaky um I've been passing, failing, passing villain for the past. Well, pretty long time, basically, um and so I looked at I started looking at the pit pressure, one um like priority bit of action ordering- and it just looks like the test- is bad. um If we look at uh and just wanted to see if everyone else improve my assessment of it, um so I'll paste the link in there to the actual problem in the test.

C

At least the problem is because I'm seeing it anyway and then I. If this is the issue, then I'll go look at some of the others, but basically um so the uh it's trying to ensure that you know it's got like two Fork bomb containers and then some innocent pod and that eviction priority zero doesn't want the uh innocent pod ever be evicted, but the PID pressure eviction is strictly By, Priority order and the innocent pod and all the other pods have the same priority.

C

So it's just random, which one gets evicted so sometimes the innocent father will get evicted first, which actually doesn't uh release or uh release many bits at all and then we'll pick the other one. So the test is just ends up being flaky and so I've just created another priority class locally and I've been running that repeatedly this morning and so far I haven't had any failures where I was getting to failure, like you know, 34 of the time without that change.

C

So as far as long tail to fix is just to give a higher priority to the innocent poet so that it will be evicted last and then won't be evicted since the uh the other eviction frees up hits.

A

You know why the block was introduced or it was never correct. I, don't.

C

Know um I needed to well yeah I, guess I need to look and see. I looked at the test, it was created in like 2021, um so I'm not sure like. Is it been flaky forever and we've just kind of gotten lucky.

A

So the story of this top is uh like when I joined more than two years back, it was all red and then uh Danielle I think made a heroic effort of making it almost green and I. Don't remember about this specific one, but uh I remember about this pressure, one, uh that's another one. So this pressure was made green, but it at some point. It will get back to red again.

A

So this one is pressure.

A

This pressure always fasting and they have another uh issue with eviction which is um I, don't remember which one is it yeah our P division also here.

A

Yeah this pressure is not here but uh I. Remember something else. It was fitting here, yeah, so I think we looked at uh I, don't remember, speed pressure ever was I.

A

Think it's the first time, but maybe my memory isn't correct, but I remember that this pressure was on continuity and something else on uh cryo, but uh I mean it's kind of important because you want to stop to be uh green, but it was not as important to somebody to like really pick it up and uh work on it and I think David get to the point when uh he realized that this pressure is likely because of change over disc size like default disc size on TI and um like we don't.

A

We just don't um create this pressure fast enough to protest to pick up oh yeah, but it's a work in theory was at that time. So maybe you discover something else. So yeah uh I was surprised to see feet pressure but yeah. If it's failing it's failing.

A

Okay, do you have a fix for that yeah.

C

Yeah give a higher priority to the innocent pilot, so it doesn't get to resolved or doesn't get evicted first, um but.

B

C

That was my assessment of it anyway, so so I was wondering like how's, it ever been passing, or has it just been random and we kind of got lucky for a while.

A

D

You get the link to that patch in there in the dock too, so I can look at it. um Yeah.

C

I haven't pushed it yet, but yeah it's just about to.

D

Er or whatever I want to see it thanks, I'm glad to see you're making things green. This is great.

C

I'll uh yeah I was just pushing the Brian channel uh I'll put the link in the dock.

B

D

Not hearing anything, are you muted over there.

C

Foreign yeah I think you're, muted, Sergey.

A

Oh sorry, did you hear what I was saying about.

D

Probably not you've been muted, since, uh since Todd spoke about putting up his link.

A

Okay, sorry about that, so what I was saying is uh this test started failing from uh yesterday, I think uh yeah I mean maybe like today morning ish, um and this test sounds like something that we may have introduced recently, because it's mere report and I know that Clayton was working on uh mirror ports and static pods. So I was wondering Ryan. Do you track this work? Do you know uh if this may be a new failure.

E

Oh, it certainly looks like a new failure from Tesco there.

A

Well, I mean new test, I guess yeah.

E

I we can look at it. um I didn't see anything merge yet into the tree into the kubernetes tree. That would impact this, but we can take it as an action item to look at it.

A

Okay and yesterday we actually merged a change from uh vinay on in place.

E

A

I wonder if I mean it's the alpha features protected by Alpha gate flag, so I wonder if uh okay, so we.

D

I remember the discussions with Renee in that it was related to mirror pods. There were some mirrorpod Discovery. He revealed.

A

Yeah, so something to take a look.

C

It looks like it's not a new test. It's been around since at least like one one, two, three.

A

Okay, this is a good news.

E

That's it merged yesterday.

E

I always suspect that that uh patch, that merge might be the problem. One.

A

So this test run at 7, 58 am today and previous one was green and it was 657.

A

That runs it open. Okay, so it's from today morning and the patch was immersed yesterday.

E

We can certainly dive into it and see.

A

Okay, do you want to take a look.

E

Yeah, can you uh gotta paste the it looks like you pasted the job yeah I'll, take it along.

A

Thank you and this flake is uh it's just a flake on device, plugin and another mirror port.

A

Mirror project determination flaked yesterday yeah it was flaking.

A

It was flaking consistently, so nothing new, at least not regression.

A

And looks like it's a new test, this one.

A

It was introduced to 39 and then it seems It's flaking.

A

Device plugged in just.

F

For the record, we have me and yeah: we have a pipe changes in the pipeline for device plug-in protest which we'll try so along the way, we'll work to improve the flickness to remove the thickness. So we need.

B

F

The tire anyway, so you can say when, but it's coming so we'll keep an eye on that.

A

I mean this one once in 128 right.

F

Yeah not earlier than 28th cycle for sure.

A

Yeah that one we discussed last time and uh I think it stopped flaking since so yeah, better.

A

Okay, uh and uh just in general, just to check on that like swap, is still failing. um Yeah. This is some flakes really random new one, but before that it was all green machinery should be good.

A

Yeah this one stopped working.

A

Yeah I, don't know how important this is. Okay, uh I think I, don't see anything uh just jumping on me.

A

A

Another mirror port.

A

I wonder if it's the same.

A

No, it's different.

A

B

It's the same as this one.

E

Is it a flake or was it.

A

E

A

E

A flake right, it's Karma, failing.

A

This one is sweet.

E

Yeah, that was definitely a play.

E

Sort of concerning um this one's with container d and the cryo one uh previously.

E

I was failing too I sort of think the patch that you mentioned yesterday that went in might be bad.

A

E

There's a failure.

A

Like flaken started earlier right, it's like I, see this flake from way past.

A

Yeah I start at the beginning, like when it was introduced, looks like a new test.

E

A

And a cryo, we do super compact.

A

Thermal failure and again yeah when it was introduced yeah before that there is no information, but then it's failing all the time.

A

No successful runs at all.

A

I think it's worth looking at yeah.

A

Eviction we looked at it of like it's flaky.

A

I'm labeled still have someone labeled.

A

Or interesting sum up uh thought it may be relevant relevant to what you changed whatever head.

C

A

Yeah I, don't know we didn't catch it because it's uh it knows labeled as not conformance or anything. uh We used to call them orphaned and now they're called unlabeled.

A

And you already have an issue for that.

B

A

B

A

Think uh this is a failures. It's uh worst looking at.

A

We want his agenda.

A

Is there a normal agenda items? uh Let's go to uh really quickly I.

A

um Anything that just touching signal, but like not direct reflection signal, so uh we wouldn't spend time here all together negatively surface. So this is uh in place pottery size which is not working very well and when they self-assign it for 128 I think oh yeah. This is when we changed everything to serial, and now it used to be returned back functional.

A

A

Google pod resources get feature. Somebody added a test.

F

Yes and I monitor ready, but yeah. You know for the graduation they're starting getting the the author is adding the end-to-end test, which is a good idea anyway, so seems legit ah yeah and they should have added the the labels. So no worries I will do that yeah. You are free to. If you want to feel free to add otherwise, I will do in namino right.

A

Now so um and I'll move it to.

A

And finally, times out with.

B

A

B

A

Is it Windows Windows yeah.

A

A

Close the result.

A

Now I'm gonna switch to bugs oh by the way yeah. Before we go to box. uh We didn't check performance uh all together lately, so let me do it before the release, so um I will already pass the test free. So it's a good time.

A

It's. uh What is this? It's a cool blood uh and it's CPU, so good, no spikes! Oh, we actually improves your memory.

A

Which is also good, we now investigate going down. He always needs to him going up. uh Let's look at the runtime.

D

I'm used to new data being on the left to me that looked bad I guess you know better than I, though wait wait. What sorry on the left and the old data on the right so I'm.

A

Just making sure that.

D

You probably know this dashboard better than I I'm. Just commenting. Are you sure that we didn't make it worse? That's my question.

A

Because, like time like, this is all data- and this is new data- okay time goes this direction. Okay,.

A

So we and something happens that we dropped uh some memory usage.

B

A

Stable and CPU yeah stable, okay.

A

Now go into box uh if you hear just for test, uh feel free to drop off we're going to backtrack.

A

A

Okay, this is just a to-do I'm, not a bug. Okay,.

A

I see many non-box here something happened.

A

Okay, that's important that I rejected by admission should be periodically I did try it foreign.

D

At me over here see y'all next week right.

A

Now static, Pro doesn't create mirror report.

A

Maybe you can, you cannot kind documentation.

A

Account tokens when Port is terminating.

A

Or interesting, please stop hook for 12 hours.

A

Oh no! It's too loud.

C

Four hours into the pre-stop hook, I'm pretty like the freestyle.

A

And I'm very interested to take a look. It even has a source code links.

A

Okay memory pressure during the button.

A

Oh, is it uh problem with configuration.

A

And, first of all, we don't use, we don't own cool bottom.

A

And I'm not sure if coupon has its own area.

C

I wonder if we just had it misconfigured.

A

Yeah, what is configured by kobada I think kubernetes.

B

A

Be misconfigured to use different Secrets.

C

It looks like a mismatch between container runtime and Cube glitter.

A

Previously, on single bottom Zone,.

A

C

Sig cluster life cycle, I guess.

B

A

What updates is there any discussions already.

A

A

A

A

B

A

If you need to keep back our label changes to fighting test.

A

Hello, but can't use import ant, 126 yeah covet was updated.

A

Yeah I wonder if it's failed on potted Mission or Port security.

A

A

Policies they have in a cluster and he didn't allow to schedule. The court I mean I kind of tell with this like.

B

A

Port is succeed, success but failed with yeah. What failed means.

A

A

What is Google configuration? That's.

F

A

The digital config option.

E

E

A

If it's interest, because if it's introduced behaviors and it's not ideal.

A

I'll keep it triaged, even though we need to read with my information will educate a priority on this.

A

Just a few left.

A

Okay, which we get is enabled, but Port is not running in here in space.

B

Cost users Falls.

A

A

It would be interesting to see Google Talks, there's a couple of talk and say: if we really picked up this configuration, it's our user error be on safe side.

A

Is owning this feature and we will create a new generation.

A

A

A

Continuity version 168 very fresh.

A

B

A

A

Plateaus I I, don't know.

C

A

C

D

B

A

Okay sounds: just tests are failing with programmer.

A

Okay, well, David is investigating, so we will triage.

A

I don't know priority, though.

A

F

A

It's new error introduced recently.

A

Okay, here we go blame.

A

Okay, I think it's a new bug.

A

I'll ask her for the.

B

A

I think it's the same as uh we just saw recently from vinay.

A

How do I find this back from the name.

A

I, don't remember.

C

Let's paste it a organization, oh you found it yeah.

A

Yeah. Thank you same.

C

Yeah I guess that issue is about if you have a static pod and you update the pod on the API server, is it synced to the running pod or not, and I thought the answer was like it shouldn't be.

A

Yeah, but what kind of oh you you mean like determination as an update, rather than update of some settings from Minecraft? Well.

C

I think like in vsk, is like a part of resource resizing.

A

Yeah I mean, but you only search the settings. There are updates like Horizons I, wonder if static code has arisen as well anyway, I will let somebody do because it may be the same update because updates from manifest.

A

I think we will preserve the hash overport, because hash doesn't depend on specific settings.

A

I will accept it, though, because it sounds like.

A

A

Make sure it's uh security multitels here, okay,.

A

As you know this here, because of.

A

Is anybody from uh Red Hat here because it's talking about cutting containers, maybe you want to take a look.

A

G

Can you yeah, can you see see me I'll I'll redirect it to so. Thank you forever.

E

uh Cc uh Joel Smith too yeah.

A

I think we just need to close it, because it's uh it may be Miss understanding.

B

Yeah, let's do it.

A

Writing his profile because.

A

He's the same as um missing process or is similar to that.

G

I'm not this eye raised, it I think it's similar to what uh Clayton is trying to do so. If you see the do prob, which is uh a worker, it checks, the status manager and uh the status manager is a possibility that this, these statuses may not be up to date. I know they should refer to the worker worker pod workers, not the status manager. So that's the work Clayton is doing and I believe. This is the same issue that he is seeing with volume manager and other other areas.

G

A

Oh yeah and it's important for graceful termination right. We didn't race exclamation one, because yeah.

G

So I have linked to his PR HPR should cover the this aspect. Also, the espr is touching multiple aspects of the com or the components that are going to use the power worker. It should also include this this particular aspect. So that's why I want to tag him.

A

uh And it is not a.

G

Regression right, no, no any other internal CI is seeing it sometimes like a one or two person times.

B

A

You have six meters left um it just conclude. Through that I can use it or I didn't move it. um I need to move.

A

Yeah this one crashed.

A

Yeah I created this yesterday as a I can remember some of one of the PRS with known potential problem.

A

Or another thing from In-Place update.

B

A

Okay, this one is the last thing that may be important.

A

Okay, so it's a feature.

A

I'm going to accept it, um it may be a feature, but it looks on a on the edge between those.

A

Okay, then, we have four minutes left. If there is anything uh anybody wants to talk about this pickup, otherwise we can go back and finish. The meeting.

A

Yeah I thought anybody like. Whenever you have, you want to talk about AWS tests. I know there are some like conversations testing and if there is anything relevant to this group, please bring it up. We will discuss okay. Thank you.

A