Kubernetes SIG Node, 10 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Node Sidecar WG 2023-01-10

Description

Meeting notes and agenda: https://docs.google.com/document/d/1E1guvFJ5KBQIGcjCrQqFywU9_cBQHRtHvjuqcVbCXvU/edit#heading=h.m8xoiv5t6qma

GMT20230110-170539_Recording_1460x1120

A

Hello, hello, it's a signals. uh Sidecar working group, meeting, uh hello, everybody!

A

It's a second meeting this year uh we kicked off last week with Matthias, and somebody else was on a call who was silent um and we've been just catching up on all open questions. We've had and agreed that we need to start Rising cap like keep writing a cab draft. So what I want to do today is to see the progress of cap draft and maybe get people um um working on piece of it uh before I. Oh, maybe I didn't I. Do it.

A

I I also asked Francesca and Swati uh about about apology manager and they haven't come up uh with specific answers yet, but I will mention what they uh what kind of problems we may have there. So, let's go through uh capdraft.

A

If you don't have a link, it's uh linked right here from around working group document, foreign, so yeah, we have small summary motivational part um I, think it was copied from previous caps. So nothing uh might uh worse, uh deep, mentioning um so goals and non-goals I edited them a little bit yesterday.

A

um So um what you want to do is uh you want uh uh sidecars to be first class decision, so we want Kubla to support sidecar pattern. uh We want a job completion, it should be solved. uh We want to mix init containers and sidecars, so you can test May, utilize, sidecars and also you want to easy injection of sidecars in any pod. So these are our goals and none goals is any other ground dependencies between containers so or um we also don't plan to support the intermination ordering at least in Alpha I.

A

Doubt we'll ever do that, but it is enough. We wouldn't.

A

um And Proposal Part: this is where we uh outline what you want to do. um So we there is a special flag on in containers that will indicate um if it's a sidecar on outside car. So today today is called Always restart policy, always um so they will not block pod completion.

A

um They will be terminated when all other containers terminate and Source contacts will also be restarted. Even when poetry start policy is never on failure.

A

um I added this line here, because I think uh I think uh Joe and Nicole I think we get to the point when we all believe that this is important scenario, uh we just need to understand how hard it will be to implement Joe I are you? Are you here, yeah yeah.

B

I I think that's right. um I need I. Just got back from break so I'm going to catch up on this one, um but yeah I should I should be able to leave this part.

A

Okay, so I think uh we can leave it here for now. Okay, so Kevin has a commentary, don't afraid to speak up. Are we okay?

A

So Kevin is saying yeah yeah, oh, go ahead. Kevin is saying that they're starting during termination makes sense to me uh in the chat.

B

Yeah I think that's right. um Tim Hawkins had a model that I kind of like to where you basically um I'm trying to remember how to phrase it right, um but you basically, you kind of make it clear what the life cycles are, so that the side cars are still kind of in an active mode and doing their normal, restarting up until correct Point. um So I'll, try and I'll try and see if I can get that into the captain in a way that makes sense.

A

Yeah, oh by the way um you you mentioned, that I I never mentioned here in proposal that uh oh, it was a complete tenure, Readiness, okay, so Readiness is defined by Readiness props, um okay, I mentioned here uh so I also wanted to minimize home kills uh by addressing ohm score like changing room score adjustment. uh Today we cope with um for adjustment, very importantly for small containers, which sidecars typically are so I I added this line here, but it may be um I'm sure how I don't I don't know if it's controversial.

A

If there are, it will be any big concerns with that. You can remove it, but I I feel like uh it can be important. So to give some um color here uh today, with sidecars are typically smaller, uh much smaller than regular containers. So when we calculate boom score adjustment, a percentage of uh nodes that are sidecars utilize is very minute like it's like it's uh less than one percent and when we incorporate um it turned out to be like 999, so it's maximum score adjustment possible.

A

So it will be like uh terminated right away uh whenever um uh situation happens and, lastly, I I think this is controversial. This is something that I'm, not sure about: uh we've been discussing the restart book of timeouts, uh adjusting them for sidecars. So today we will, um when we restart containers who keep increasing timeout interval between the stars and all the way to up five minutes and five minutes, maybe too long for sidecars, because they want to be alive all the time.

A

uh So the question is: do you want to do something with this timeouts.

A

C

There might be a difference too, on crash restarting when it's starting up versus like through its entire life cycle.

A

Oh okay, what's the difference like.

C

When it, when it first starts up, it might be restarting because it can't pull an image or something because so haven't draw out a little bit longer. To start up, the Pod might be okay, but while after it fully starts, maybe then it has the the quicker restart time or something.

D

C

Think thinking that makes sense, yeah.

D

That makes sense and also like, if it's not configured correctly during startup, it can.

C

D

B

Does this need to be? Does this need to be something? That's really special about side cars, or is this kind of painting that we lack proper configurability for containers in general, like I'm, a little worried about making side cars like have too many, like special cases versus just trying to make General cases sufficient that you can use them for sidecars the.

C

The um killer, setting kind of felt like that too, like maybe there's other containers that would want to adjust that.

D

A

Think simplest.

D

Yeah, okay, I think simplest to begin with, is not to change anything and then see how it is and then make changes. So if you propose too much it'll be hard to reach an agreement. Okay,.

A

Let me remove this part, um because uh this part was also I added this section about uh ways to abuse this.

A

So yeah um uh we'll get to that, but um how users may start up using it and the user may decide to use sidecars as a way to run regular containers and if there is a special restarted back off timeout policies that they want. They made start using it for all the containers and it's not what we want to uh encourage.

A

B

Yeah I agree with that. I also think it's if you, if you start making too many things like just slightly different than um it's gonna, seem kind of magic to people. Why like why? One container mode is like just behaving different than another container mode, whereas if we try and keep the behavior uniform, um it solves the problem you just mentioned before, which is people won't switch modes for one of the side effects but then also I, think it'll just be less surprising, open, cleaner.

A

Yeah I do think that um word always than confusion, because always here indicate that uh I mean to your point that uh we want to minimize amount of magic um always has already known word. uh It means something, and it means something specific for regular containers. uh It doesn't mean that they will be restarted during termination, for instance during Pro termination.

A

Do you feel that uh this may be yeah.

B

I'm worried about that um I think that's a really good point. um The way that we were hoping to phrase that is, that um first side, cars, the way I'm hoping to do this and I don't know if I can achieve this. But what I'd really love to do is is make it clear that the sidecar's life cycle isn't in the terminating phase.

B

Yet, even though kind of like the pot as a whole is- and so because of that, you would expect that it's a restart policy to be applied normally, um so they will require some explanation, but hopefully, at least like given the explanation will be clear that it's like still in a mode where we still make sense foreign.

A

For naming um I don't know if anybody wants to uh take this and like ah fill this up anyway, um so user stories I think we already have user stories on the top uh with uh status, mesh and logging containers. uh We may want to add user stories here and I wasn't sure what kind of user stories will be useful here. If you have had suggestions, uh please speak up. Otherwise we can just remove it.

B

Yeah I'll do a pass too I know there was a couple people that were complaining when I was talking about termination that they didn't think that sidecars were Justified At All by what I was proposing, but um so we need to make sure we have maybe it's already. There are the cases we need to make sure we have a use case that makes it clear that, like being in the init phases, means that you can have like one side. Car start before another side car.

B

If there's a dependency um and I think there's a couple other cases, so I'll review them and make not to that.

A

Okay, thank you.

A

So uh in the risk and mitigation sections I um wanted to have this, uh how customers may have used this feature? The first example I have, if usually decide to use sidecars to run regular containers and I mean this is just a brain dump and I want to brainstorm on this music as well.

A

So if you have an idea how to abuse, uh please pick up and I just put different scenarios, how potentially they can abuse, but for something else, I can't even come up with a way how it may make life uh different difficult, so how, for instance, May usually Implement some Noisy Neighbor scenario with that or may Implement something very strange based on this new Behavior.

A

So first is uh having the most side. Effects is sidecars as a way to run regular containers. One scenario I make I came off is how it can be useful. Is users may decide to implement uh kind of timeout logic? So imagine you have a one container that just run asleep and terminates after sleep completes and all the other containers are running some job as a sidecar containers. So now you, if you effectively implemented some implemented um uh job execution with uh like maximum timeout, um so this is.

A

uh This is a pattern that we cannot Implement today easily, uh but with sidecars customers may decide to start using it um and it worries me a little bit. So it's a kind of an abuse of uh sidecars um sidecars feature. What do you think.

E

I, don't see the bad part of it, I mean at the end. It doesn't put any workload in danger. No.

B

um I think it's worth the listing uh I! Don't think that one, for me at least, is like a show stopper, because I could imagine you could have like a parent process that also like was kind of like a watchdog and timed out, and that's also probably no better than what you have here. So I suspect that you could try and find a way to do it today and you probably wouldn't like it better than the sidecar approach.

B

E

B

E

We were sort of in it inside the container yeah.

A

Okay uh and yeah and I listed this, uh um so if we will Implement somebody started back off, timeouts difference or some other difference that will May benefit users and they may start abusing it. So that's another reason why not to implement special features.

A

Okay, balloon side: cars there's another feature, um it's again just uh brainstorming here, so what users can do that they couldn't do before is uh to allocate some large portion of resources like CPU, for instance, and pin some CPUs in the very beginning of initialization stage um and then like they can not specify real resources uh for uh regular containers.

A

So they can minimize those resources for regular containers because they just reuse whatever this balloon container allocated in the beginning, I can't come up with what's wrong with that approach, um but it's something that people wasn't able to implement before.

A

A

Okay, nobody sees the problem. Okay, I also, don't see a problem I.

D

Just wanted not a blocker yeah, just yeah yeah, but.

A

I I- and this is a scenario here because of I- think I thought a lot about like how you need containers and regular containers. Reuse resources, especially keeping in mind topology manager and the CPU manager um and Francesca promised to team up and Swati, came back with some answers.

A

B

Yeah I mean I think this is a good list.

B

um I'll try and add anything I think of too um I think the interesting thing about what we're doing is like people were already doing side cars just in another way, and so, unless we actually make the situation worse um then like, if you can argue that we made it strictly better than um that's I think we're in pretty good shape for most cases, I I am a little worried about like people wanting to do complex ordering and things, but even there, I'm, like I'm, still think we're probably better than they were before.

A

Okay, yeah, uh do you want me? uh You said you like my list. You probably read through all of it, but let me finish: okay, just uh for completeness uh sidecar that never becomes ready. It's similar with you need containers that never complete, so I. Don't think there is a new problem, uh intentional failing of or terminating sidecars. uh We will abuse Kubota a little bit because it will keep restarting it even for jobs. It may be unexpected, uh but I don't think there is a problem with that.

A

I also started try to think of different ways: customers May abuse our Eventing system. So let's say you have a job and you want to send some events back to kublet, so you have a sidecar that determinates and termination will be translated into event back to um back to cook CTL. So you can try to come up with some understanding. What's going on, based on termination of sidecars and restarting of sidecars but yeah. This is uh such a like a long shot, so I didn't release it here.

A

um And then keep inside car alive. uh Somebody edited just now right, yeah.

C

That was me I. Don't know why it's not having my name on there.

A

um So I don't think sidecars can keep job alive. It is quite opposite. It will terminate the.

C

Job right, it's something we have to consider. You know we may want, based on the use case above with logging, you might want to be able to hold termination up for a while, so that you can make sure all the logs get logged out, but that same kind of thing could be abused, potentially by one user, keeping their pot alive way too long. Okay, so.

A

There's something to consider.

C

Yeah yeah on termination.

B

We're a little bit limited there because there's like the termination grace period and some other things that um I think from an API perspective would be pretty hard for us to violate um so, but we can, we can think about like I I'm, assuming as long as you extend that far enough that you within those bounds, you could kind of get more time.

B

It's a little tricky because I'm expecting a lot of sidecars to get injected by like an admission, controller or admission web hook or something and in those cases like I, don't know if you should really be extending somebody's determination. Grace period when you put the web, the the sidecar.

C

On and it's hard to tell the user from the web hook at that point too.

B

Yeah, it all looks the same. That's a really good point. um This does bring up. Another thing, I was thinking which is. Are there? Are there risks that we introduce that exist only because people have already been creating sidecars today and have binaries that work a sidecar binaries? Even though there is no built-in sidecar mechanism and they might migrate these to this new approach and then get some kind of Behavioral change, they didn't expect.

B

I wanted to think that one through like say like istio or whatever any mesh anybody they've already got side cars and they just kind of naively start migrating to this new thing. um Are there any kind of like rough edges? That would be like super surprising for them.

A

Yeah, if there is no comments on the call, it also comes back into backward compatibility issues um because uh it's one of those issues. When you take some assumptions and then you try to apply these assumptions to something that doesn't have the same behavior amen.

A

I think it will be minimal risk because sidecars typically wants to germinate themselves on other containers. Terminating, so I think they will be.

A

B

Nothing happens.

A

Earlier then, we will terminate them.

B

Yeah yeah I'm not super worried about it. um Yeah.

A

Or maybe one option: maybe they will start germinating themselves, but we will restart them because it's too early for us and this determination station will report. So we will restart them and they will think that they just starting up uh so that maybe.

B

Yeah I mean that could have some yeah I will have to I might actually go through and look at like a couple concrete cases, because ones that are doing metrics might like be registering metrics versus yeah I. Don't know, I'll have to think that through.

E

But I think as long as we document it correctly, people should be aware about what will happen if they they do some edge cases like this. No.

B

Yeah I think documentation is probably like. Oh we're, ultimately going to be able to do um if they're, if we do identify something, it's like really weird then like. Maybe we can give specific guidance for that particular case, but I actually suspect that we might not find anything in your book. We should um um you know John, Howard or anybody working on istio or any of the other main ones, maybe like a uh early adopter can hopefully give us a good feedback.

A

So another um again, aesthetic things that I were thinking about is: uh if customers will set liveliness probe, that uh defined on side carbide actually pinging the job, then you kind of don't terminate the job, but you get a signal about liveness, Behavior or job. So today, yeah I, don't know I, don't know how how even to express it. But uh what can be the risk here? But I I know this.

A

um You can get a two containers and one of the containers have a live in this prop and the sliveness prop not necessarily being in support opened by this container. So you can ping report of different container and then, by doing that, you I mean you may get a signal about loudness of a job without terminating the job, but I think there are better ways to do that today. So I don't see it's a big abuse.

B

Yeah that sounds right.

A

Yeah just trying to spark conversation and some ideas. What other situation may happen? Okay, um then we have record compatibility. We will have upgrade and downgrade scenarios covered in prr, pure production. Residence review um outside of that I was trying to come up with the ways how existing code may break like existing third party code, so we will fix all the bugs in kublet and in the control plane, but outside of them, um so some assumptions that people may take on ports today that will break with sidecars.

A

So one assumption is, uh nothing is running in the port when any container starts. So if we need containers somehow like enumerates our processes or relies on the fact that there's only like um they only writing this like uh themselves, so they may break because of sidecars um I'm, not sure if it's um significant risk, just one of the behavior that may break for customers and if they rely on that they will be with broken I.

A

Think the only way around it is to document strings and say like if you're doing that you will will break your assumption.

A

um Second, one is uh obvious: incorrect equivalent resource usage. So if people don't pay attention all the new Fields, so they just they don't parse it as a type A strongly typed object, but they just like only look at few they're interested in. They can miss a new field added to init containers, let's restart policy, and they will calculate the resource using correctly.

C

Yeah I've got a Prometheus, Dash or grafana dashboard. That would definitely have to be modified for that.

E

A

I think it's inevitable with our design we will come up with. uh It will be: oh yeah, so sorry uh it will be a similacitation.

A

And lastly, if there is a controller that strips out all everything they don't know about, then they'll run in the problem, because if there is three power, three start policy always from minute containers and they will render this proton usable uh I, don't know if anybody ever doing that, but one of these guys came up is.

A

Okay, I also definitely not doing.

C

It I think most of the.

C

I think most of the modifying web hooks I think the API is to patch the existing thing. Not here is the whole thing back modified, so that might not be that big of a risk.

A

Yeah I mean, hopefully is it I mean yeah I think doesn't support new Fields. Then it may be a problem.

C

But I'm thinking the common way of actually doing that might prevent that problem.

E

Yeah, okay, maybe some open policy agent rules could reject the the pod.

C

C

Yeah I can see that.

A

Okay, so they failed to parse it. They feel they crash with the error and the seller searched as a pot Beyond scheduleable yeah, it's it is a Brisk. Yes,.

A

um Yeah, okay, I think record compatibility is covered. uh Is uh all the assumptions I think we just need to document all those situations and behaviors um shouldn't be a big problem and operating downgrade section. I haven't completed it yet so if somebody wants to take ownership, uh that would be great I. Think Cobalt will will reject new Fields I I'm, pretty sure it will, but I didn't check. So maybe we need to check if existing couplets needs to be modified, so they will not accept.

A

um What is a this new restart policy always.

A

Okay, a recirculation for scaling and Cloud Mission, um so I put this uh formulation today we calculate the resources maximum of Maximum for unit containers and sum of containers. um This is uh requested forward. Formula uh more complex formula will be like if we just add a sidecar containers here to formally will be like this Max heat cut, there's plus, assuming that all sidecar contains running at the same time, uh and we also add the sidecar containers to all the regular containers and then take maximum of this one and this one.

A

But this formula is not completely correct. uh Actual formula should be more complex, like imagine, you have a init container. uh The use of the seed containers will be this container. It resources itself, plus all the sidecars that started before the Senate container.

A

um So you need to go through each of them like sequentially and like calculate this formula and then take marks of it, and then you also need to sum up all sidecars to all regular containers and they take marks of these two numbers. So.

C

Yeah, we need to make sure that we somehow expose this through the Prometheus metrics, in a way that this can be reproduced.

C

uh So that users can track it, I I've been doing this kind of calculation on pods already and it's kind of painful. Maybe we just export the actual final result, since it's getting more and more complicated.

B

It's a nice idea.

A

You think this method needs to be exposed by scheduler or complete.

C

I'm not sure where the metrics are exposed now.

A

Hey I know how to do it from coupons: I'm, not sure how uh control plane can expose it. Yeah.

C

But it's it's common to scrape all the cubelets, so maybe maybe that's the right place to do it. I don't know.

E

B

So the schedule you're going to need it anyways or is it going to be able to get by without knowing.

C

That uh no it actually it will need to know it if it's uh weighing it against Dakota yeah. If.

B

C

If it's, it's, not necessarily the scheduler, but the API server somewhere for a quota needs to handle it right.

A

Yeah I'm I'm actually still not clear, not completely clear on how scheduler calculates it does it calculate uh final use when you need to installation stage completed already, or it still allocates all the memory in case both will be restarted and needs to run through the installation. Again, it.

C

Reserves now it reserves the max of all the containers and the sum of the the regular containers. Whatever is the largest of those against the quota, at least okay.

A

So I assumes that it may start again.

A

That's fair, okay, so we don't need to have more complex Logics than that uh yeah. But it's a good point about formula uh explosions is a number.

A

Cpu managers uh here, as I said I I, pinked, sweating Francesca to look at court and yeah, maybe I just started looking myself as well. um The complication here is how to like. You already have problems with reusing resources after you need containers completed uh because, especially when continuous demand full CPU or they want to, pin CPU somehow, then it may be in substitution. We have a problem that containers cannot be scattered after initialization stage is completed because there is no more resources anyway.

A

uh That's how we need to be cleared out, and um especially with this complex formula when you need continues, um you'll have different set of sidecars scheduled already. We need to understand how like allocation and alignment of all the uh CPUs will work with topology manager, I.

C

A

C

Be able to simplify it and just assume that when it's successful all of the side curves will be running. At the same time,.

C

Right take the max that it will ever hit and reserve it rather than well it. It only got to that index.

A

Right, yeah, I I. Mostly what are the situations when uh you have a scope of port for uh pneumonode allocation, so you want to run a whole port and one Numa node um and then so there might.

C

Be two different function or calculations, and there might be the one on the Node versus what's used for quota.

A

Yeah this is like uh topology and CPU manager is a they put Advanced um kind of hints on how to schedule uh specific containers on specific CPUs uh and that's where I hope there is no complexity. I, just uh don't don't really know how it works.

C

Yeah I don't know either.

A

D

A

I think this is all we have today. uh We still have whole PR section to complete and test plan.

C

A

C

Problem up at the top I don't know if it really fits, but something to maybe talk about.

D

A

A

D

A

B

B

Yeah, it should be possible with one I would I would be a little bummed out if somebody had to have like one init container to do the first pull and then another one to do. The watch yeah.

E

Yeah I think how she go. People would be happy with this, but.

B

You got the probes right, so you should be able to like indicate like. Oh hey, I've got my first pull ready. Now you may resume initialization.

A

Oh yeah, I didn't mention uh termination anywhere, so I think an implementation.

B

A

You can get.

B

uh I'll do a pass in the next couple days and focus first on termination. Then I'll do more General passes.

B

Yeah, you can add a add a section.

B

This week is definitely cup week for me.

A

Okay, uh so we have test plan and upgrade download strategy.

A

All this uh do.

B

We want I, don't want to be labor, but do we want to list out some of the Alternatives that have been considered and rejected somewhere in this I I feel like that, might help us stay out of double jeopardy with people that are like. Why aren't you doing X we're like? Oh, we thought this like there's been like in iterations of sidecars and there's a ton of things that we've we've seen and tossed out. In fact, we've got I think a big list on our our our meeting.

B

Our meeting notes of some of the rejected ones um I can help fill that out too, but.

A

B

Do it because I already have a document for that? Oh awesome, that's even better.

A

Think to myself.

A

um Do you want to take any section here, maybe prr or.

E

I I could check the upgrade download a downgrade if you want. Yes, thank you.

B

Maybe version skill with those usually go together.

E

Okay, yeah: what's the difference version skew is when you you have like a new couplet uh and an old control plane or the opposite or.

B

E

B

Okay, that's the main one: we're worried about.

A

This I don't know your email, so I can't assign it as a comment, but you probably remember: okay and test plan. I think uh that may be a good, uh oh yeah, a good thing to list.

A

To this some test cases to demonstrate that we know what we're doing um many times it was um I mean all other attempts to do. Sidecars were always ejected because of lack of testing that we we assumed, um and maybe it will be a good idea to list all the tests that we want to implement.

A

I can start filling that up and then we can go by example. Sounds.

B

Good yeah I'm willing to do a review, Paris and add anything after somebody said the first one.

A

Okay and uh Joe: it's.

B

A

B

A

Perfect I think uh that will be quite enough.

A

Is there anything missing.

E

B

E

The document that we should register ourselves for the the cap list in signaled.

A

Yeah we will do that uh I'm, mostly thinking about API review, like uh thinking of a time when we can start presenting this document to people like Jordan or somebody else, so I think. Maybe if you complete these few sections by end of week, we can like next week uh we can start doing it. Thank.

B

You um I think that's a really good idea. I've noticed they like to see it early I mean we already have basically a proposal for what the main API teaches so I think um I. Think that's a good plan like regardless of what we achieved like we should go on to their next schedule and just tell them like hey. This is how we're thinking.

D

B

um That kind of gets them ready. Okay, I've done that in the past, they seem to like it especially early in the cycle when they're less busy to get a chance to kind of like get their heads around something, and that makes it easier for them later.

B

A

um Let's uh try like my next meeting for sure. We need to have uh document ready to share I agree. Well, if you have time earlier, it's even better because we can have previews. Thank you uh anything else. People want to discuss today.

A

If not, then we can get uh 17 Min at 15 minutes back. Thank you. Everybody bye-bye.

B

Thanks take care thanks, bye.