Kubernetes SIG Arch - KEP Reading Club, 21 Jun 2021

Previous Meeting

⏯

youtube image

►

From YouTube: [21st June 2021] SIG Arch - KEP Reading Club Community Meeting

Description

Kubernetes Enhancement Proposal (KEP) Reading Club is an initiative by sig-architecture.

KEPs covered in this session:
1. https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/2599-minreadyseconds-for-statefulsets
2. https://github.com/kubernetes/enhancements/blob/master/keps/sig-windows/2258-node-service-log-viewer/README.md

A

Okay, so before we get started since there are a few, uh since there are folks who are new here, the way these this usually goes is uh we take one cap. We spend around 10 to 12 minutes on that one cap reading it and then post that we start discussion on it and uh ask questions if any and related things so for this particular cap, I've also linked the pr that was made for the skip to be merged.

A

So uh if, after reading the cap, if possible, we can go through that and see what the process usually looks like. So I'm going to start a timer for around 10 minutes post that, if, like.

B

C

A

Time then always we can extend it, but I'm going to set a timer for 10 minutes and we can guess that just give me one. Second, okay and the timer starts in three two one yeah. It's.

A

A

C

A

A

D

Dale sorry, I'm late, I thought I'd jump in if there's a meeting going on.

E

Thanks tim, can you see the history of the chat? Do I need to paste you which one we're on.

D

Yeah, no, I cannot see the history okay.

F

uh We are looking at the stateful set, adding a min.

F

F

Yeah ravi has to drop at the bottom of the hour. So let's get started.

F

Ravi, are you there? Yes,.

G

So uh what are the next steps here like? Do? I need to walk through the cap or something.

F

You can't hear me we can hear you, we can't see you. I see, okay, right, ah perfect,.

G

Okay yeah. So uh what what are the next steps like share my screen and then walk through the cap?.

F

Sure, usually, I encourage people to like tell a story like what is the problem, and why are you trying to? What is it that we are trying to do here, like, from a user perspective, sure.

G

Let me start sharing.

G

This all right, uh so this is the cap at a high level. What we want to do is we want to add a particular field called min ready seconds to stateful set the problem.

G

As of now uh that we're trying to solve is say if a stateful set part comes up, it is always deemed to be ready and ready in the sense that it is ready to serve the request, whereas in in real world it may not be ready, it is still reconciling the state, for example, if it's a stateful set like cassandra or some other database, we would like to have some time for the new replica or the new part that is coming up to reconcile so that it can properly serve the requests that are coming in.

G

So we have uh this particular field already available in other workflow controllers like deployment and demonstrate, but we do not have something similar for a stateful set and as far as the real world applications go, I think I gave one example. The other example could be a particular program which is behind the load balancer any software, which is behind a load balance. We do not. We want to wait certain amount of time before it actually gets taken out of the load balancer.

G

So these are like some of the scenarios that we have faced within uh openshift and we noticed that this could be beneficial to the community as well. So we started working on this feature upstream in 122 timeframe. This has been around for a long time like I think people have been asking for it, and someone has, I think, mayank has started working on other aspect of uh this particular feature. We have something called max unavailable as well, which is not yet available in the stateful set.

G

uh If you look, if you look at the linked kubernetes issue that it tries to solve, it has got both the fields. uh This particular kept solves one aspect of it, which is related to the min uh ready seconds.

G

uh So those two are the technical problems, and uh this is just a bit of background uh unless someone wants to add uh something.

F

Yeah one question I had right off: the bat was like how how does this relate to all the probes and all the things that we have for probes.

G

Yeah, so that's a good question and I think we have mentioned that in kubernetes document as well, uh so the readiness probe or the liveness or the health checks they will give you certain amount of time before which a part can be deemed ready. But that does not guarantee that it is ready to take the request in the sense that you can have an additional delay which says, I would like to make sure that the pod is available for a certain amount of time before it is deemed as available.

G

So the containers, all the containers within the pod should be ready for that much amount of time. The the liveness checks or the health checks are at the container level like technically, uh whereas the all the containers in the pot should be ready. For this much amount of time is what mean 30 seconds.

F

Last question: uh so when does this specific amount of time start? Is it after the other probes are done and they give a thumbs up or before or during.

G

The first one is done by the cubelet uh in the sense that those health checks are done by cubelet. After that say, the container is ready and if all the containers in the pod are ready for that much amount of time, then it would be considered for then it would be considered as available. So the first level of checking is done at the cubicle.

D

So after the last container becomes ready, then it starts this timer.

G

Then it starts this time so that all the parts are in one x amount of time.

F

I think we might have to add more uh color to the to the cap.

D

So yeah I mean, I think, that's a useful distinction. um My I have a similar question um if I can jump in um this says, making it uh ready but or available sorry but available in whose opinion like do. We prevent the uh end points from being marked ready or are they visible as ready in the normal fashion? But we don't move on to the next item. If we're doing like a stateful set update or something.

G

uh The update is at the replica level, uh it's it's it's not at the container level, but it's at the ripple. So right once we do uh once we check for that particular plot to be available for x amount of time we would go on, which is part of the.

D

uh So um I'm coming up sorry, I can't hear if you're still going no go ahead.

D

Sorry, I'm looking at this from the from like the networking, because you mentioned in here load balancers, but I'm not sure I see the integration point, because a load balancer will proceed as soon as the pod is marked ready and there is no additional, like ready, plus wait a little while longer in all the load balancer controllers.

D

So this additional weight either blocks the actual setting of readiness like the pod readiness probes does or it doesn't. But this doc doesn't say which one.

G

So uh I'm not sure about all the load balancers, but uh the way I was or the way we were thinking about at least an open shift perspective is uh we would like to use uh or the pot to be available for that amount of time, which includes the mean ready seconds as well like say in the template. If you specify mean 30 seconds as like 10 seconds or 15 seconds in the load balancer configuration, we would increase the value by 30 seconds by 10 seconds or 20 seconds.

G

What about the the amount of value that we give.

F

So from reading here ravi, it seems like the only controller that is going to be using this information is the stateful step controller and nothing else right tim. Does that help uh what you were thinking about.

D

I mean yes, except it calls out somewhere in here. uh Let me see if I thought I saw.

D

Yeah here, as a kubernetes user, I want to leverage stateful sets min ready seconds behind service load balancers in cloud. This doesn't cover that at all, because the cloud load monsters don't pay any attention to min ready seconds.

G

Right so that has to be manual like there is no automatic way at this point of time. We would add that uh additional amount manually is what we have who, who would add it manually. So when uh like when openshift gets installed, it actually creates the load balance raspberry and the right time would include the min ready seconds that we have for those components like it's.

D

Load balancers map to services and services map to pods, and we don't know whether there's a stateful set in between like there's there's the chain of trust, doesn't include the workload controller. That's off to the side.

G

That is, that is right. So that's why it is manual at this point of time.

D

Okay, but is there a plan? It is the is the long-term goal of this to factor into the rest of load. Balancers.

G

That is the plan, but we at this point of time we wanted to make it manual, but if we get something back that yes, we like this to be included, we can do it like, for us being static, is good enough at this point of time,.

D

um So this is the fun part right like I understand that it's good enough for you, but I'm not sure that I want to add api that doesn't do much unless it does stuff for everybody else like. I know it's really minor, it's like one field right um and it wasn't actually clear if there's any controller changes other than uh work than the stateful set controller right.

D

um But if we're going to add an api I'd like to solve it to I'd like to make it worthwhile to make it make it hold its own weight right like as an alternative question, why would we not push this all the way down to pods and then make the cubelet not set the ready until after all, the other ready signals have been set? Plus this time.

G

D

Then all the controllers get it for free, correct.

G

So that is something mentioned in the cap as well like. uh If you look at the issue, the the open issue, I think uh brian grant, clayton and the rest of the folks were interested in having it within the power, but there was not much of effort uh in that direction or someone has traded tried it in the past, but did not go through because of some technical reasons that I am not fully aware of, or I cannot fully remember at this point of time.

G

Okay, here we go. Oh wow, it's from 2018, okay yeah. It has been around for a long time uh and there it was mentioned that we wanted to have it within the pod spec, so that everyone can actually use it.

G

But in one of the points in the nonviolence we have explicitly mentioned that it takes a lot of change and we want to have consistency for all the controllers. That was the reason we are pushing it in this direction, but say in future. We are like. All of us are fine. Having uh or moving in the direction we can. We can do it.

D

Funny that two years ago, brian suggested the same thing that I just came up with uh sorry, I'm reading I'm skimming the issue now.

D

The problem, okay, so I'm I agree completely with what brian said in his may 31 2019 comment, which was, I also think min ready seconds should be added to pod, but I'm not opposed to adding it to stateful set since other controllers have it. Basically, the precedent is strong and I would not be against adding it. What I would question, though, is if we, if we set it in the controller in sorry in stateful set, and we later wanted to add it to pod.

D

How do we not make those, because I don't think additive is what we would want.

G

So my understanding is, it could not be uh like we would remove completely from the stateful aspect and we will have it within the.

D

Well, we can't remove stuff, so once it's added it's forever right, and so we have to define how it works with the new thing.

G

D

Like imagine, I have a stateful set where I specify in my pod template the min ready seconds and in my stateful set min ready seconds. If I set them both to 30 seconds the obvious behavior. Is it's going to wait 30 seconds for cubelet to set it? Then it's going to wait. Another 30 seconds for the workload.

G

Yeah, so my understanding is safe, it reaches the pod spec that will take precedence and whatever be specified, say if you cannot remove the api. uh The value that we have in the pod spec will take precedence uh when compared to the value that we have within the workflow.

G

But there is something that we did not think through here like how uh how the migration would work.

G

Does that answer.

D

I mean yes, it's not it's not a satisfying answer, but it's an answer. That is true.

D

I don't want to monopolize the floor. Anybody else have thoughts. I'm going to read this comment. I'm.

F

Just thinking that this has been approved for 122 and seems like this needs an update ravi with the discussion that we've been having.

G

Sure and uh to answer your other question, I think even the deployment controller does the same like we do not have end points, controller, getting updated or getting involved. uh It's just the state, it's just the deployment status that is getting reflected with available graphic details and the main available uh seconds. So I do not know if it needs a change across the board. uh Just like uh the 30 seconds within the pod spec.

D

Yeah, I'm just reading the comment linked and it doesn't. I don't think it's a particular problem in that it's bringing parody if this was net new I'd probably push harder against it. um So I won't, I won't get in the way. um I do feel like it's a it's a half of a solution.

G

D

It's the same half that deployment has so.

G

So uh like we are trying to promote it to beta next uh release. Perhaps we can have this discussion during uh the beta again and then.

F

Yeah, but I do want to capture what we talked about today and uh update the cap, so we don't have to like try to remember. Where did we talk about this, and when did we talk about this, and do we have notes, and do we have recording um just remember brian's comment was very cool right now right, you know when tim was reading, brian's comments, and you know he came up with the same thing.

F

So it helps if we, if we update the cap right away uh with the discussion from today, so we don't have to uh relitigate one more time sure.

G

I can update the cap like I'll open a pr with the discussion that we have and perhaps comment on that.

F

Yeah absolutely sounds good. Thank you. Yeah! No.

G

D

Yeah sorry mostly, I just would like a few words that, like this doesn't automatically handle load balancers and it's you know. That is a separate issue. This really is just bringing parity with the other workload controllers. That's the main objective, and if I had seen that as reading it, I probably would have just not even commented.

G

Okay, yeah in the normal section we have mentioned that uh we wanted to bring it uh on par with the rest of the controllers, but I think I can make it explicit in the comments yeah.

D

No, no, I saw that and then I saw the load balancer parts and my brain went. uh Oh these things don't make sense to me.

G

Okay, I'll update the load balancer at least awesome, uh so I need to head out now thanks a lot for having me. Unless you have any questions, just let me know, or you can ping me on slack.

F

Yeah we can follow up on the thread uh ravi on sick architecture, I'm sure uh the.

H

F

The rest of the folks will have some comments. Sure.

G

Thank you. Take care, bye.

F

That was good. We actually found some stuff.

D

What I found the last time, the last time we did this reading group is, I would bring caps that I thought were slam, dunk, sure things, and then we would still find things to talk about.

D

And I mean that in a good way,.

F

Yeah, of course, amado back to you.

A

Okay, uh so before we start, I just had one question like in general about the process, so in this particular cap I saw a field called uh required monitoring. Metrics, so is that is that something that's part of the template and required by all caps, or is that like um per cap or it's okay? If that particular cap doesn't require it or do features need to have some form of monitoring provided in their implementation?.

D

I think it is an expectation unless you can explain why it's not.

F

Necessary also, is it coming from the prr section right.

A

I don't think so. I think it was like a separate uh field. Let me just crush it.

I

Yeah it's under the prr, it's a sub heading: okay, okay,.

F

So basically, the point here is: when you put this feature in production, you you need to get some metrics out. You need to figure out like how people are using it uh things like that right. So we are trying to look from an operator point of view and not just like the developer point of view, which we typically end up doing so that that's the reason why we we added the prr section and then we are making sure that people are thinking about it.

F

Clearly and saying um you know yes or no for for things and exactly what they would be doing to make it more friendlier to operators.

F

A

Thanks, okay, so if no one else has any questions, I guess we can start with the second gap.

A

So this was this: one is on the note service log viewer. Again, I'm just going to set a ten minute timer and we can get started so the timer starts in three two and start.

E

And a bunch of people did the little green check mark reaction in zoom. So if you finish early feel free to toss that on and we don't have to wait the whole.

E

F

Mother vido how to use google for timer for two minutes or ten.

A

uh Yeah, so I just type in a 10 minute: timer.

F

Okay, yeah next time you can just share that screen. So we know how much more, in a minute sure.

A

Okay, uh two minutes are remaining for what.

I

Also for shorter gaps, I think, like 10 minutes, seems a bit too long.

A

B

D

Are we all ready.

F

uh Supriya's still has a minute.

E

H

Oh, I have completed it. I still don't see that yes mark on my.

F

uh Click on reactions and you will see a check uh green check.

H

Okay got it. Thank you.

D

On zoom web, it's hidden under participants.

D

Or am I the only person who still has to use the web interface.

I

D

Has to work will not let me install zoom.

D

Even in a docker container, I asked.

E

I

Shall we start if everyone is done? I think yeah.

D

So I cheated I pre. I've read this one before.

D

Is one of the cap owners here that wants to talk about it or we just talking about it.

A

I don't think one of the cap any.

E

A

Either of the capoeira, I tried contacting someone from sick windows, but I don't think they could make it.

I

B

Have a question: oh sorry, go ahead! No! No! Please.

I

I wanted to ask like, since this was not mentioned in the gap, then there probably might be a way of doing this, and I don't know of it, but like the cap mentions that, like currently administrators have to ssh into it, and instead of that they can use cube ctl, but they did not mention how authenticating in that case, would work like how would cube ctl know that this person is allowed to use the logs command.

D

It says somewhere in here, uh there's an r back a new rbac scope for node logs as opposed to pod logs.

D

So you can grant our back to node slash logs as a sub resource.

F

Sorry, who's beeping.

D

I don't hear you beeping, I think it's you.

F

I'm the only one getting beeping. Sorry: okay, stop.

E

So I had a few questions that I wrote down. I see there's a giant discussion though, so I didn't read all the comments, so this might have been discussed. I'll rely on youtube to answer those, probably um the control plane. Logs specifically, I know cloud providers lock down access to the the control plane.

E

um Would there be any? Is there any restrictions or sort of discussion around pulling the logs from the control plane? There.

D

um I think that is a great question that isn't clearly covered here um in the case of managed control planes. Does this require that you give access to those? um I I don't think the answer. I think the answer is no right. um As in you, you know, look at a gke cluster, you don't see the master it it's not there right, it's not a node it. It might be running cubelet or it might not be.

D

I mean it is, but it might not be right, um and so I don't think, there's a uh a limitation on that. It's not. They are not control, plane, nodes, they're, just an abstract control plane. So I think that could be clarified here. That's actually a really good point.

E

Yeah, that's the first thing my mind jumped to was huh managed nodes. I wonder how that works. Okay, um the next one was um the specific callout for system d was really interesting. um Was that talked about for limiting the scope to just systemd anywhere.

D

The it's funny I find myself as the representative of this, because I was asking these same questions um is: is somebody taking notes that we can forward on? Are we capturing this other than the recording? Oh.

A

Yeah so yeah, uh I'm I'm writing down a few notes. Awesome.

D

Thank you. um So the initial draft that I read if my memory serves me, was very journal. Cuddle focused um and the other things were there. In fact, I think it was originally a flag. You had to say whether you wanted something from the journal or from a file and the heuristic that is now in there was sort of, at my behest um to say I bet most of the time. You don't need to specify the difference right so only leave. It only specify the difference if you have to so.

D

The heuristic is first we'll look in the journal, then we'll look in some commonly named files um and we'll probably be able to find it and it if we make it through alpha, and it turns out that that's not true, then we can come back and reduce it.

F

B

F

One related observation I had was this heavily relies on our cubelet, our version of cubelet, so people have other cubelets too, so um you know how do we end up putting this in the conformance program? uh That would be my uh two sense on it.

D

And is the heuristic, is the heuristic part of conformance or is it simply like, given a you know, deterministically crafted query: does it work right.

F

Right and what should it actually return? Should it return exactly what our cube returns or it could just return garbage right so.

D

Yes, um and if I compile on the same question, one that's not clear to me is the requirement for interleaving across nodes is that is that going to be client side node like how is that done? Given that you know time, stamps are not always equally accessible, given different logging formats right, if I'm pulling from varlog somethingsomething.log like it, may not even have time stamps how? How or am I going to? Are we going to sort of loose parse those.

F

Yeah, this seems very red hat centric. To be honest,.

D

It is, and now the the the thing that I was sort of convinced of as I read this is almost everybody's using systemd anyway, and you know my own experience aligns with that as much as I'm not so happy about it. Oh this is recorded, um it is, it is, seems to be the state of the world. So that's probably okay, um and if we end up well we're gonna go through an alpha. If it turns out to be a big problem for a lot of people, we'll have to revisit it.

D

That was my feeling.

F

I agreed I the one thing I really like about: this is it'll help windows, folks, uh people who want to poke into windows and like get locked back, makes their life easier without having to you know, go to your infrastructure and go pull stuff yeah. I definitely like that aspect of it.

D

One of the things that I disliked um about this is because cube cuddle logs was sort of a terminal command before now it becomes a both a branch and a terminal um it it's unclear whether those flags are additive. Like the principle, I think that cobra espouses is only the the flags add, as you traverse the the tree right, and so all the flags that apply to cube cuddle logs, don't apply to cube cuddle logs nodes.

F

D

And in fact, some of them are defined differently, they're defined as the same name, and they have different defaults or different meanings.

E

That's a lot of what I wrote down.

D

E

A game control maintainer right now was just I I looked. I saw mache signed off on this, so the there's breaking changes here, where, if you have a pod named nodes, what happens to the sub command.

D

E

There's there's definitely some incompatibility here. I actually probably would have wanted to see this as cube cuddle like node logs as its own command, because, having worked with cobra, it's really hard to introduce a sub command to a command that already exists. It's like really difficult to pull that off correctly. So um you.

D

E

D

Yeah you should you should jump on that. I didn't I've, never used cobra in great detail outside of being trivial stuff, um one of the things. Maybe it makes sense to have a like a dash resource or something flag to it, but that was my first thought, but then you know: there's flags here that are defined with different defaults and having the default set differently for different resource types is weird.

D

I understand the desire not to have more top level cube, cuddle commands right like in hindsight. Probably this should have always been cube, cuddle logs pod right, but it's not and uh I'm not a cube, cuddle maintainer. So that's on you guys to figure out what you want to do forward. um Nor am I a great cli designer so.

E

Yeah, I'm far from that too. But I'll bring this up with six cli at our next meeting and um yeah. We may need a couple changes in here too for compatibility all right.

D

That's great, I didn't think about the case of what, if I have a pod called nodes.

F

It almost feels like don't worry about the things in the middle there's. No api change. We are gonna. Do some tweaks at the back. We're gonna do some tweaks in the front and everything will be fine.

F

Don't worry too much about it. Magic is happening. That's the feeling I get from this one.

D

Like I, I noticed that uh cute cuddle logs has a dash tail flag and this has a dash tail flag, but they have different defaults.

D

They seem to mean the same thing, but one is defaulted to negative one, and one is defaulted to zero um and uh cube cuddle logs has a follow, but this doesn't uh and there was not one more. What was the other one? I lost track of it.

D

There's a few other flags that cute pedal logs has that this doesn't.

D

So yeah I mean my main comment would be it'd, be nice to unify these into a coherent story.

C

I have a separate uber question as in.

D

Already looks like you're talking, but I hope there, you are sorry just delay.

C

Yeah, so general cuddle is actually very verbose, or rather there are too many events there. So if you have a cluster with 20 nodes- and you just get all of the logs of general kettle on your master, is there some sort of threatening throttling involved? Or do you just overwhelm everything that's available.

C

It's very good. I mean I really like looking at general catalogs, but I really redirect it to a file and it's usually a few hundred megs and then I very gradually begin to look at it on an individual note. So if I have.

D

This has a dash service flag, so I imagine you always want to specify the dash service flag, though it does seem like it's optional.

F

So do we end up ddosing the api server.

C

F

People run this at the same time and there's huge amounts of logs coming back through the api server.

C

So the only grace is that the system admin can run is the only one who can run it. So hopefully he knows what he's doing he or she, but.

I

Also, one thing which was mentioned in the risks was what, if, like the logs, which we are getting, they are too big in size, and uh it was written in the second line that to mitigate this, we can document that node lock should always be rotated. What is rotating the logs mean? I didn't understand that.

D

In this case context, rotating generally means throwing away old data, so you don't have last week's data. You only have yesterday's data.

I

So would that mean that if we are to support this, uh there would be a change in how much data we store? Past data restored.

D

I don't think that's going to fly like we can recommend that people do anything we want, but doesn't make them do it. We need to protect the system against it. Yeah.

F

Typically, the g, you gzip the older thing and then start a fresh file right. So that's what they call in quotation. You still have the old data. It's just gzip, so you'll have to go, get it for yourself. If you need it.

D

Which I believe journal cuddle does anyway right right or journal.

C

D, but uh yes, uh there are companies like splunk, who are built for essentially handling very large logs and general. Kettle is one of the logs of that space. It is not.

F

Yeah, that was.

C

The other angle I was thinking.

F

About was um lots of these uh log aggregator type companies and they end up doing side cars and to capture this kind of information. So are we like competing with them to some extent, uh are they doing something better that we need to learn from.

H

I feel like they're just doing the log forwarding right there, it's similar to how we're doing like locks off any parts.

H

So, even if we provide this option, it will just forward this to splunk, and I think it would be helpful for teams where there are newcomers and they don't have to give permissions to all these node level. Permissions.

F

As an operator like, would you like this like? What's your experience.

H

I think I would like this so uh so in my team, like um so, we actually have like a month or two to see the people like um how they are. Can we trust them, and then we provide access. So for us. If we have this option, we could just let people read from splunk and we don't even have to worry about the access controls. So that's something that I was interested in as soon as I saw this.

D

It's clear that this is never going to be and should not try to be a replacement for splunk right. We should not provide a very rich query language on top of this. This is not elasticsearch.

C

And the other thing was about the time skew which tim was talking about. So what are the thoughts there? Do we actually specify which node is providing? What and.

D

I don't think that's covered in in here very.

I

I

I've got to drop now bye. Everyone. Thank you for doing this.

D

I

B

Forward to the.

I

A

Thank you thanks. Everyone thanks.

D

For this, thank you eddie. Are you going to go back and follow up with these folks.

E

Yeah, I put it on our 60 agenda.

D

Cool one of the things that wasn't clear to me also sorry last parting, note all the stuff like grep and interleaving. It wasn't clear to me whether that's being done client-side cubelet side or api server in the middle. So it would be interesting to figure out if they're expecting the cli to do all that heavy lifting. How much you know buffer does it need to store in memory or is it going to buffer to disk or if we're expecting the cubelet.

E

To do all that all right, I wrote that down.

D

A