Kubernetes Data Protection Working Group, 24 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Data Protection Workgroup Meeting 20210324

Description

Kubernetes Data Protection Workgroup Meeting - 24 March 2021

Meeting Notes/Agenda: -

Find out more about the Data Protection WG here: https://github.com/kubernetes/community/tree/master/wg-data-protection

Moderator: Xiangqian Yu (Google)

A

A

Morning again, uh today is wednesday march 24. Fourth, this is the kubernetes data protection working group meeting. Let me share my screen. First, apologies uh today, majorly we have two major agendas and then we'll do the leave to an open discussion.

A

The first thing is that uh shin and I have been working on this whole container notified cap for a little while uh today, I'm going to give you they give the group a more kind of look back, plus current status updates. uh This will take. Probably 30 minutes of the session depends on how many questions we have and the second agenda is. We will talk about a little bit about the white paper. Was that without updates, then we will be opening the uh opening the session to the audience.

A

If there's any question or anything you want to discuss, please feel free to bring it up. There.

A

All right, let's get started, let me bring up my.

A

A

A

So uh you probably have been hearing this topic for a little while just to give those who are not very familiar with this topic a little bit background, so that in total, two attempts so far to define a mechanism to send a command to an container in kubernetes world. The first attempt the campus list over there. It was crd-based uh controller- will reconcile on the crd and it needs to be a privileged part because it needs to run arbitrary, cube, exact comment against arbitrary part and container, and it mainly was targeting and solving the application.

A

Consistency snapshot problem. That was the first attempt and uh there were a couple of concerns around that. uh One biggest thing is about the security, uh because the controller needs to be privileged.

A

uh There were good amount of pushback from signal and uh and also as api reviewers talking about the possibility of moving this into a different direction. The other thing is that this seems to be a good there's. Some there's there are other use cases which can fit into this model and they want to extend the scope a little bit so that use cases like sending signals to a set of containers or parts in the kubernetes cluster.

A

uh uh When, uh for example, if you want to debug, you send the signal to you know, adjust your log level and from seek node there's another thing called uh when they try to tint a node or mark a node as unusable and scheduled.

A

They want to kind of notify all the parts that are running on that particular node to to let them know if there's anything they need to handle should handle this right now before they mark the note down.

A

So uh those are the cases then I went ahead and proposed this container notifier uh it was co-api based uh cooperate will run the command against the pod and the container. Eventually uh this, basically is the main purpose is not to widen.

A

The scope of pre-reached controllers that offered by the community cooperate already have all the necessary access to run. Those commands against any arbitrary part and container on that particular node, and that is also to support uh generic more generic to support other use cases, and this is a second attempt, but we received a good amount of feedback from the second attempt.

A

So we had a couple more rounds of discussion. Then we had a second and third attempt so deeply in this topic. What are the major concerns in this thing? It are three aspects.

A

The first thing it's scalability, so cupid is it's already a very heavy, but you know undoubtedly it is really really important uh core kubernetes component. How do we? How do we make sure we don't.

A

We don't send too much traffic to the api server from kubernetes and how do we ensure kubrick can actually handle uh so many notifications when the pod can be in the order of thousands in the cluster?

A

The second thing that drove a lot of discussion is the whole imperative versus declarative concept. Kubernetes apis are mostly, uh if not all, declarative right. This is this whole base level triggering versus age of triggering right. uh Everything is about level triggering kubernetes work and signal uh execution hooks. I tend to be more imperial. Basically, you send something and you leave it there that you're wrong.

A

This there's not too much need in terms of retrying logic like a regular kubernetes controller will be doing in case of failure, uh taking the execution hook for application consistency exam for as an example we even if we want to hr, I would retry or maybe we don't even want to retry- to limit the the quietest period of a certain application and while, during application, consistency snapshot process, if we introduce this retry logic, may we maybe we can still bound it, uh but amit it will extend the period or the sla we need to requires.

A

The application in case of failures signals is another good example as well. The third thing is that there are some interesting use cases where this whole deterministic, sequencing or ordering of execution is desired.

A

uh Yeah application requires enquires for sure, and when you send a signal, for example, if you want to temporarily adjust the lock level to debug, you don't want to do this. You know you don't want to leave it there forever, because that probably will you know, write too much block into your system and uh make your disk full very soon.

A

So uh how to deal with the newly created or newly deleted part in a kubernetes world, it remains a problem to be solved. Basically, uh pods can go and get the again go and come at any time in this in the kubernetes environment, even nodes, so uh how we process that is another major discussion, topic question so far.

A

No, I then I will move on so so we have two attempts in the second attempt that this is the first attempt in the second attempt the call thing, so the idea is: have a inlined container loaded, notifier, a list of actually container notifiers in the container core type, and this is not. This is not an api object, it's just a united type, so uh the structure of the notifier is pretty straightforward.

A

It's basically just a handler which does the command and the name, and the name uniquely identifies uh uniquely identifies a specific handle to run in a container you're going to have multiple notifiers.

A

Users specify this container notifies during the part during uptime or container creation time. So when a user wants to trigger a specific container notifier, they create a notification to trigger to request the a specific container of notify by specifying the name, the notify name.

A

You can see the notification spec structure in the right side right hand, side, the second one on top of that, they also specify which part to be notified.

A

So there's a part selector as the first field groups, all the parts that codifies that has the label uh and uh uh get all the and find the corresponding notify within those part definitions and they send the coupe exit command.

A

So you can imagine you have uh let's say uh my sequel deployment somewhere with the leader in the forward enabled so uh when you do an application requires logic and application consistency snapshot.

A

What are you going to do is that, okay, you select other parts, that label with, let's say, mysql and then uh there's a notifier defined in the container say acquires me whatever it is, and then you send the signal to that corresponding notifier and the results will be recorded in the notification status.

A

uh You can take, you can see that the notification status- the first thing it has- it is a list of pod notification status and each part notification status contains potentially multiple container notification status, because each part can have multiple containers and all these containers can have that notifier defined this making sense. So far.

A

So the major concern on this design there are a couple of aspects. One aspect is there could be thousands of parts in your cluster.

A

How do you scale uh your notification status because your path selector can be arbitrary right? It can be anything and then it may end up thousands of pot notification status nested, underneath the net notification status.

A

First of all, it will create a good amount of pressure on api server and also kubernetes in the future, because it needs to process every single of the resources, every single part that qualifies the selector.

A

Secondly, the notifications status will group so big such that it breached the api service limit the size limit and you can no longer store all these status in the api server. So that's the very first concern.

A

The second thing is: this is basically from jordan. uh Liquid.

A

His concern was that go ahead.

B

A question: if you don't mind what is in the what is in the status? Is it just like the output of the the command that got run.

A

uh In the notification container notification status, it contains couple fields; originally, it was whether it is succeeded or not at what time it is succeeded and it is corresponding container name container identifier, okay,.

B

I I'm looking at this design and trying to understand how it's different than just using cube cuddle exec with some sort of an rbac scheme where the user.

C

Who defines the.

B

Pod basically gives a white list of commands that can be exact like it feels similar to that in terms of what it achieves am. I misunderstanding some crucial notion, or some crucial portion of this.

A

Sure, though, the you mean the security piece is that what you're talking about I'm.

B

Wondering like if, if we just like cube, cuddle exec lets, you run commands and the reason we don't like it is because it's just you know, if you don't, if you don't uh put any limit on the commands, you can run, people can use it to perform attacks. So this mechanism lets the guy who defines the pod sort of enumerate a set of commands that can be run from you know from other controllers.

B

Doesn't that function just like a.

D

Functionalized, it is the same, but.

B

D

There is a like security aspect of this right and also adding this as a poor definition uh makes it a part of uh the kubernetes. The core object is just more convenient yeah but functionalize your right. It is the same, achieve the same goal.

B

B

For scaling, we have a solution to the cube, cuddle exec uh issue right like where, if I wanted to run cube, cuddle exec on a thousand pods.

E

B

I would just do that and the I don't actually know how the api server stores the output of all those commands, but presumably it it uh it does it in a scalable way. um So, like. I wonder if there's a way to just leverage what exec is already doing and then have this.

D

But you don't really: where do you uh save your you're? Not really saving those in your ikea objects right the results when you run that, where are you saving this, so you need to have your own controller, take care of those right here. We are talking about we're, seeing that in an api object status. So that's the concern.

B

D

B

Wondering if that's the right, because it seems to create a scaling problem so like maybe we only need to store the stuff that is lightweight that cube cuddle exec would have stored if you had run it.

D

Well, q code well cube cutter does not save it. You have to save it somewhere in your controller. You need to create your own, maybe your own crd, to save that somewhere right. So you actually have the same problem and just uh really your controller is handling in some way.

B

Yeah, I don't know mechanically how how exact well.

D

Exact, doesn't really it just. It returns the results you you can run that from command line, returns the results and then who is saving that you're running on command line? That's just down there and command line, but you're going to.

B

Have a controller.

D

Then your controller needs to save it somewhere right. So that's what I'm asking: what are you comparing with right, the status update here we're talking about? If this is part of a status in an api object, then uh there is concerns uh from the api update point of view. There's a.

B

We can currently the client is talking to the rest. Server cube api server from some so.

D

What we are talking about here is not like running the command itself, but it's after the after you run in the command. There are some results coming back, and then you need to save that somewhere. So tell me that if you're using it while you're saving today.

B

No, I I'm saying I I don't know the answer to that. I'm sorry.

D

So so we're talking about, I mean we're talking about different things here right. So if you.

E

D

E

D

B

Why can't we yeah yeah so.

D

What I'm saying is we're talking about it. We want to save that in an api object, it's not going to be saved somewhere when you run that that's you run that on your cli right you, you see that that's the output right, your controller.

B

To save the api server is getting the data from the from the node to the to the rest client, and it's acting as it's.

D

Fine we're talking about what we're talking about that results has to get somewhere in the api object. That's what we're talking about here when you're using pubecaddle, I mean it's not really saved. Okay, unless you have a controller that.

F

B

Way so we are not really talking about the same thing. I believe that it is saved on the node no.

D

G

Container, no, it is not sure, um hey um just want to intervene. um I think this is ali from redhead. um I did a little uh deep dive into how cube cuddle uses um implements log and exec and from my um study.

G

Cubecut.Exec opens a bi-directional connection from api server into the cubelet uh into the uh container. So if the api server is just proxy uh that connection um so.

D

How many containers.

G

Are you how many.

D

Containers are you going to log into at one time when you run that comment once.

G

Yeah, um so that's a valid question. All I'm saying is to your point. um It just opens a bi-directional connection and it does not store that matter.

B

Okay, so so thank you I I suspected it was something like that, and I know that, for the logs they actually are stored on the nodes and kubernetes has a way to go, pull them out when you want them.

A

But right logs I will be there that I I yeah.

D

But it's not in the api object that it's not that easy for you to go! Go past the logs! That's not what what we but.

A

I I okay go ahead. Please.

D

So so, if you're, using cube and also what I'm saying is uh we're talking about scaling issue right when you're running that cube color command you're running one one command, so you are logged into one container at a time right here, we're talking about we're doing a program. Then, when we select a lot of uh we use the part, the the selector, then it could be thousands part and we're running that at one time. That's a concern. Well,.

B

That's what I'm trying to get to is that we're we're creating a new problem, because, because this is a new we're running a command sort of non-interactively with cube cuddle exec, you would just get the output beam directly to you when it's never stored but like here.

D

We need to yeah, but we need to get the results right, but you can yeah, so you can achieve all of this by using cube cuddle. Yes, but then you still have to figure out like how to store that yourself right so right today, I think, like uh the backup uh vendors, some some of them they well, I think most of them should be already doing this, um so they have to handle that themselves. Maybe actually we should probably ask someone to answer this one other than we talk about it.

B

You don't actually need to to store the app, but you just need to receive it and process it and then make it and then determine whether.

D

Right right, so we need together. We need to get that result somewhere right when you get it, I mean so so.

E

The brother, I'm not.

B

D

B

Criticize the designs yeah.

D

I'm just trying to understand I'm just saying what I'm trying to say is that the the problem is uh um the potent potentially you could have thousands of parts. So I think the concern is um the scalability. I think that probably actually exists today. If, uh if you are doing a uh if your class and then you actually have to require thousands of parties, you also have that problem.

D

It's just it's not, maybe not an aka update, and I don't know your controller- maybe handle it differently, but you should have some similar uh scaling problem. Actually.

A

D

A

D

So we are talking about that was saying. The scalability problem is already resolved by uh cube cuddle. I'm saying it.

B

Does not, it has the same problem, but maybe okay.

D

So maybe actually tom- I don't know saint tom- you guys are already doing this. I think it might be better, for you actually uh give an example, uh because otherwise, uh maybe we're actually going into different directions, but actually it's the same. Similar problem.

H

Yeah so the way we we tackle this is we the um cuddle, exec libraries, return, three values, standard or standard out and error code. If there's an error code, we treat.

B

That as an error and.

H

Then update the status in whatever you know, whatever crd or api object, we return back and the actual um standard iron standard out. We we log in the controller um you know because it doesn't actually show up in the pod log itself.

H

So then we just we essentially stream all the data back to the pod and then put it there.

D

So what what what happens, if you happen to have like a thousand parts that you have to acquiesce.

H

D

You have typically.

H

We don't see a thousand pods because we kind of tackle you know individual applications right. So a few spaces.

D

What is the typical, what is the typical size like what is the typical like or the largest, that you see, yeah applications.

H

You know is up to, I would say, like 40 pods.

D

H

um But you know average is much less because usually it's like kind of a you know, some like mongodb or something where you have to go and execute in different.

D

Right so probably in practice, probably this is not really a problem. Maybe.

A

No, no, it's not it it you in practice. It is right.

D

So I'm talking about the quiet case so maybe.

A

D

Since we are making this more generic, like signal.

A

D

A

In that case, then, you can say that we are enlarged.

D

Gene is a problem because we.

A

Are making this engineering only solving the application is not a problem. I don't think scalability is really a concern because, as tom was suggesting normally you won't have over, let's say a dozen of parts for a specific uh application that we need to requires. But if we're talking about signals, then this is a good chance. We will bridge the thousands uh we're getting to the southern's uh kind of domain.

A

Tommy was saying something right.

H

um For the signal case, you won't have that much output being sent back, though right I mean, because you're essentially just running a single, a single command that won't have too much output. Unless what output are you thinking, you'll have to collect for from sending a signal to a thousand pods sure.

A

Even you, even we make the container notification uh structure as concise as possible. If you do, if I didn't do the math roundly only if you contain let's say 16 bytes of id, it will explode if you, if you get like 5000 parts.

A

A

And I think this is a valid concern in many cases, so we let's see- let's see we have some kind of you know solution. Let's see whether we can solve this problem, uh but I want to discuss with the plane offline a little bit and to get where his idea is because I'm interested to learn about that ben I'll ping. You later on.

B

I I I don't have.

D

Are you, is it clear now, ben? Maybe you know after tom explained, maybe it's now more clear.

B

A

I don't think he's not arguing about that. She was, he was saying. Are there alternative ways that we can save this status? We can not using this status at all. I don't think.

B

That I was trying to figure out if there was something else that we could leverage and why this feels like a new problem when, fundamentally, what we're doing is something that people already do um and it's- and I think we understand now it's because we're not in a position to directly stream the results of these commands, so someone has to buffer them somewhere um and that that's where the problem comes from yeah.

B

D

Think it's the existing problem. It's just.

B

Have we considered a model where there's like an object per pod? So, instead of having a giant array, you have just a bunch of objects, and so you scale by.

D

That leads to your next slide, maybe.

A

You you did, you read my slides, oh yeah,.

A

Okay: uh okay, that's the first attempt, so we collect this feedback right did. I did I talk about the other feedback we have seen like. uh I think right. I.

D

Think you have covered it.

A

Well, the odd question.

I

So here, basically, we are when we create notification to request triggering corresponding containers. It's really cuba that figures that right.

A

In container cube right, uh-huh, yes,.

I

Back to the previous discussion, I think basically, the problem here is this: has the potential you know for people to use it? However? They want so outside of you know, snapshot use case and quieting people can use this like ansible, you know just to run commands at a bunch of parts. You know which is can be more frequent than running application, consistent snapshots or you know, as far as scalability, I think it's same as you know the kubernetes model, where people just write custom controllers in their pods, you know and they're watching and updating crs.

I

So I think what this enables is that people don't have to write custom controllers. They can just abuse. This mechanism, to you know, run commands across a bunch of pause, look kind of like answerable. You know you just unleash the command on a bunch of nodes. I think this. This is this. This is where scalability can get affected.

I

People use it for other purposes. That I mean it's generic, so we wanted to use it people, let people use it. However, they want. So I think that's where the problem is coming. The fact that it's now.

A

That's that's. Basically, that's basically the point yeah they were trying to make. Yes again, if it's purely for application snapshot use cases, I don't think the scalability problem is there because the application owners will be the ones defining the part selector.

A

There will also be the ones who label the parts the way they want it, so that the correct parts are selected and there will be also the ones defining the execution hook command. So if this is a purely uh kind of you know, application consistency, snapshot use case, then I would be less concerned about the scalability, but extend this to a more generic way opens the door at least.

I

And how does this proposal solve the cubelet being the bottleneck problem.

A

uh Oh, can you repeat the question? Okay in.

I

Your previous slide, you had said. Basically, cuba is already bloated. It does too much and.

A

No yeah this this proposal brings that problem.

A

This proposal brings that concern because kubernetes is already kind of busy, so uh it is possible that many parts are scheduled on a note so having coupled to run all these notifications in once, you may may I'm saying not necessarily for sure but may uh bring them. You know too much burden to the cuprate.

I

It doesn't solve the problem yeah. That's I want to make sure okay, yeah.

A

We bring the concern.

I

A

So uh we discussed it and shin, and I we had a couple of rounds of discussion with the api, reviewers, etc, and then we come up with the second attempt, so the container notify paste doesn't change, so it's still gonna be in line, uh but instead we introduce this part notifications back as ben was suggesting just one to one mapping right and this part notification status will contain container notification status again.

A

A pod can have multiple containers that match a specific notifier and they also it will not only contain the status but also have some high level part level. Aggregation results like completion time start time succeed or not.

A

I will explain why we need this slide later so result you can imagine, is you create a pod notification and all the results will be stored in the pod notification status, and this limits the part number to one in this specific part notification, then, on top of that, you have this notification spec again, it also have a selector majorly.

A

This thing is for user friendly cases, especially for signal you don't want to manually, send every single signal to every single part using part notification define that bob, the dice lana can actually scale and also high level controllers may utilize this one. But on top of that we add a couple of things, one or two things. Majorly. One thing is this whole policy thing uh it defines one. Should this part this selector should cut the scope so right now there were two possibilities: one, it's all parts.

A

Basically, the the controller will keep on reconcile on this notification object if a new part get added, it will be claimed by the controller uh under the old parts policy and getting a part notification will be created for that part, and the other policy, for example, is called pre-existing parts only.

A

It means that at the creation time of your notification, this is the time that you market in the api server for your notification object, use that timestamp to find any existing parts that is up and running which have which have a smaller timestamp than the notification creation. Time way, any newly created parts after the creation of the notification will not have a pod notification created.

A

uh That's basically, this is majorly to solve the problem of having a fixed scope and the ensure ordering of the execution uh for commands impaired, especially for the quiet and enquiries case and there's some signal cases.

A

The another field we add is called parallelism. If you're familiar with the drops or batch jobs api, you will find that the parallelism similar mechanism over there. This is majorly targeted at avoiding users from creation, creating a bunch of jobs. Let's say a thousand jobs to be kicked off at the same time which crashed the system.

A

It has a limit of how many concurrent ongoing part notifications you can have for this notification object.

A

uh There's a question in sorry. I can't, I can't see you raising hand etc. uh Please just speak up.

D

There's a question: what about deleted parts.

A

Okay, uh question about delete parts there are, there are two things to to think about.

A

One thing is that, if you think about uh in this mechanic, then the that, if a part has been deleted before the notification is created, so it's not up and running there's no problem.

A

The problem happens when the notification is created and they already sent a pod notification to a pot and the part executed, and then they don't get deleted. Then let us get deleted in this case.

A

If e, either the part notification will fail or it will succeed right, there's the the result should be deterministic, but the problem is in the in the paired execution case where the first part, a notification actually succeeded for choirs, they say using fs, freeze and the second before the second one is send the part get deleted. Then your file system may be locked in this case. uh I have to be honest with you. We don't have a solution for that.

A

uh This is. uh This is something we still uh we're still thinking uh at this moment.

D

Yeah, so that's easy, then we should uh just fail uh well, depending on what, if we are doing class, then we should just fail that choice right because otherwise, what's the point no.

A

No, it is possible at the quietest time. The part is still up and running right and then, uh if you imagine a lot of level controllers say, oh now requires my application. No, no! I get exited execute it. You go ahead and do your volume snapshot or volume backup or your application, backup whatever it is. It takes some time right before you execute inquires against the part the could delete it.

D

A

And then your part notification will not reach.

D

Yeah that will fail. Basically that's going to fail. We still need to run on quest, but then this whole thing will.

E

Fail should you fail right.

A

uh No, you can't, you cannot fail because it's deleted, so it's not in your scope.

D

But then, okay, I'm not sure how okay I'm not sure what this means. So I think.

F

I think I would, I think what I was getting at is like. Are we going to take ownership, then of the pod uh when we start running notifiers.

A

That's a great question, so basically, what you were trying to say is that you maybe put a finalizer on the pod saying that do not did it et cetera, that's a good question, but how do we know this yeah.

D

I don't think we are in the position of doing that. That's just too much overhead for us.

A

It's not only overhead, but also, how do we know there's an upcoming comment right, we're talking about the commands in pairs right, uh which will have this kind of you know, split brain problem uh where the universe is not is different.

A

I did have a different proposal before, which is to have a recorded list of parts that have a specific part notification executed.

A

In that case, I can imagine the the the way the a viable approach, upper level controller creates and notification sees a notification object and it creates pod notifications right and a part notification to parts are one-to-one mapping they're one-to-one mapping. So that's can. Let's say this is a choice and when you do an inquiries, you should be expecting exactly the same universe of your quiet comment.

A

I I guess we need to leave some kind of this called nation logic to upper level controllers. From this notification specific level we probably won't be able to solve that problem.

A

Did I answer your question? Surely uh I think so? Yeah: okay, uh uh the last piece is that notification status will no longer hold a jargon list of pod notifications and container notifications results, but rather it offers high level aggregates uh the completion. Time and start time are both kind of optional. In this case, the completion time will only be marked if the policy is pre-existing parts. Only imagine the word. If it's all parts, you will never have a completion time. Basically.

A

Okay, uh that's basically the uh current ap as the second attempt of this. uh The second attempt.

C

I had a question about the all pods policy. Then you say that it doesn't.

C

We never record a completion time then, uh but does that mean that that, in effect is a notification policy, that's in effect forever, so that any new pilots that come in that notification spec will immediately take effect.

A

That's correct so.

C

The indication is that the the application- that's triggering the notification, will at some point, decide enough's enough and delete the notification spec.

A

That's correct uh by saying that, if you say all pods until the notification is deleted, any newly added parts which match the label selector will be notified.

C

Okay, thanks for the clarification, no problem.

I

So one question: um so in phase one, we talked about the pod notification object, which is used to basically um specify the pos we want to notify and then, in this phase, we're adding a new abstraction called notification right on top of pod notification to cover different parts right.

I

So we are basically just making scaling problem.

I

More kind of worse because we're not expanding it beyond just a specific label, you expand it to multiple, like labels across different parts. Right I mean I understand, like you're, not controlling it, but specifying the parallelism, but you know users can still configure it. However, they want.

A

No uh there's a key difference over here right. The first difference is this: whole uh notification.

A

Controller logic will not be encrypted right. So in this case, cooper will be not the ones, selecting the path to be executed. That's what the other one and then you know it. At least you know somehow shift the concern from sick nodes to uh more or less upper level controller. Where this is, I don't know yet. That's one. The second one is the aggregated results. Is it's recorded instead of detailed ones in the upper level?

A

Notification object, so path, notification will have a per part status and notification will own all the part notifications created out of it and then, if anything wants to see a detailed drill down, they can always get it. That's the second thing, so there's no.

I

My point was that doesn't reduce the load on api server, so in fact.

A

You're, adding because.

I

Now we're also adding the aggregates to whatever we were capturing before.

A

I I agree, but there's a difference between a user.

A

uh You know shoot themselves in not sure themselves, but a user intentionally to break the system versus whether we provide a mechanism to them to prevent that from happening right. The parallelism mechanism is there to prevent it from happening.

I

I mean if the parallelism is something that can get controlled by the admins of the cluster. Then yes, but if users can specify them, then he wouldn't do much. You know, because.

A

Yes, it's the same thing as jobs right if a user can create jobs, it's the same thing you you you can as long as you have the namespace access, you can create thousands of jobs. At the same time, I guess the the mckinsen is very similar over here you, as a user of your cluster.

A

uh You can always kind of you know, find a way to you just screw it up. I guess.

I

I mean, I guess, the the way we currently control, that is, that we have coders per name space, so we can, for example, prevent the user from creating too many parts or too many pvcs or you know, launch a denial of service attack. But here, if users can set their parallelism field to arbitrary large values, then nothing can.

I

I mean basically we're just hoping you know um the other coders would kick in, and you know, for example, the part code would kick in and prevent you know. Okay, that's.

A

All yeah, due to maybe this, will help relieve some of your concern is. There was also a discussion around whether we can we should set a up limit for the parallelism by basically providing a wrench that is enforced by the controller.

A

The range can be some number like, for example, 100 or something like that. But this is you know it's really hard to decide what number exactly we want to put over. There basically put a a a bar or a maxed number of notifications that you can spawn from a notification object.

A

uh But fundamentally you're right: yes, parallelism only gives them the mechanism. It doesn't prevent.

I

And just to clarify so here you've been you're, saying we, it will not be in cubelet if we're still relying on cubelets for the pod notification updates, we're just not relying on cubelet for the aggregate notification updates right, you're.

A

A new controller.

I

A

You are absolutely right, so we imagine this to have multiple cases right, so uh we imagine the application controller most likely. It is reasonable to for them to use the part notification directly so that they can control the universe for choirs. You have this universe for enquires, you use, you have another universe.

A

They don't necessarily need to rely on the notification to achieve that, and the notification, then, uh is a more suitable way of kind of selecting large, relatively large scope of parts to run, let's say a command or signal, as in the signal et cetera- and this is more like a user-friendly thing uh again to uh to the the point that uh previously brought about by uh shirley.

A

This is uh this is basically uh for for the application, consistent snapshot, use cases. I think it's still better, that the uh application controller can have fun granted control over which parts to notify and which part to not nullify.

A

Anyway, no problem anyway, we're at 54 51 minutes uh mark. So any more questions on that by the way the cap are listed and I already shared the slides all the links are there. So, if you're interested, please take a look.

B

I had a question about um so obviously the the notifiers have to be at the container level, because different containers might want to run different commands in response to notification.

B

Have we thought through? If, if notifications should always be at the pod level, or if it should be valid to like notify just one container of a pod um or like how do we decide that granularity, whether to do it at the pot or the container level.

A

uh First of all, the command has to be executed against the container a specific content right, that's one, so the uh defining the notifi notifiers in the container level. That makes sense right. Yes, yes, so your question is.

B

Why do you go ahead? The notification should that also be at the container level, or should the notification be at the pod level.

A

The second attempt it's in the part level so that.

B

A

A

Okay, uh let me let me give you an example: why they, first of all there, there could be multiple containers within the pot right. They they run. They do different things.

A

However, it's totally possible that they all have same nullifiers defined.

A

Especially for the signal case right, if you define in the the container level, you still need and identify to find which container it is within the pot. So you need the part name anyways. You need a container name anyways.

B

Well, but in this design you don't need the container name.

A

Right, you can define the container level, you don't need the container name. You only need the part name and notifier name.

B

Right well, I guess that this proposal makes it like if, if I have two containers in my pod- and they both have a notifier with the same name and then I want to run a notification, I I have no way to run it on only one of the containers, I'm forced to run it on both and that's by design.

A

Oh, that's a good question. I I honestly haven't.

D

That is actually by design. So why don't we share the cap? Actually, we have some uh comments. There.

A

D

uh The the cap for the content notifier, we actually have that explained notifiers the name of the content notified the trigger. um So it's basically just based on that name. If you.

A

What is the I, I didn't know that.

D

Just just to go to the the cap itself, just go to your files, go to cap itself uh when it explains what is in the content notifier uh that act over there. This is the api definition. Actually we'll have some examination there.

D

Right so uh yeah, so this is a container right, so we're adding this kind of notifier and then uh in each continent, notifier, um so each kind of notifier. We have this. uh We have this name, uh so this name um right. So this one so the basically the container these kind of content notify in the container must have a unique name. And then you move down move down a little bit. uh I think it's.

A

I don't think we just got this.

D

Oh no, no, it is it is that it's actually it's basically um uh well. I need to I need to check this one. I haven't had to look at this one for some time.

F

D

No way yeah, so there is no way to differentiate that then.

B

D

B

But but why like like you're, making something impossible that that we could easily make possible? And I don't see these.

D

Okay, so maybe you can either proposal on suggesting on how to how you want to uh differentiate that.

B

We would just we would just add an optional container name field, that if you specified it, it would only notify that one right, yeah.

D

But that's not it's yeah, normally it's possible, but normally. I think uh that pattern.

A

Let me, let me think it through that that I don't want to have. I have you, I have to be honest with you. I haven't thought through that that.

C

The the two different containers from just using two different uh notification names if the intent is for a notification to be delivered to a specific container for a specific reason. Does it make sense that that both the containers would use the same name for the same reason,.

B

Right, there's both there's both use cases. You may want the ability to just broadcast a message to all of your containers and all of your pods, in which case the convenient thing to do, is define your notification with the same name on all of your containers. But then you may have a different use case where you want to just do a one-off and the the fact that you can't target one container forces you to like make a second notification that runs the same command with a different name or something lame like that.

B

F

um Yeah, I also have another question. I think, um while we're talking about this it, if I have, I can have three containers and each container has a container and a photo with the same name would create a pod, a notification. Spec that says go run this notifier in that pod and I would just get a success or failure.

F

Could it fail for one container and pass for the other two containers you can we.

A

Can completely fail on some of them and succeed on some of them? That's really why we have a list of the results.

F

Do we have a sorry, I maybe I missed it? I thought the notification status was just a.

A

F

It's a list: okay, nevermind. I apologize.

A

No problem: it's here see the status is a container notification status, but ben's point. I I think it's a very, very important idea. The typical case I can think of is you have multiple parts in and sorry multiple containers in the pot. Maybe some of them is the main container you don't want to mess up with, but you want to adjust local levels with other containers.

A

That's that's that that seems to be a legit case to me, but I would think it through. Okay, I would discuss with shin offline.

A

All right, uh we are close to 58 minutes mark, so uh we probably won't have enough time to discuss the uh white paper thing. I guess we just think of flying scene.

D

Yeah, that's fine.

A

All right uh any questions.

H

F

H

um Container notifier, you know how's it kept going. Is it still under review? You need more.

A

That's a good question, so uh I'm from my side, I'm pushing api reviewers to take a look at it, because this this involves a lot of change shin and I already got uh at least the bare minimum consensus from most of the major reviewers.

A

uh We just you know, need to push them to do the work and sign it up, but uh I I need to take part of the plan, because I was not having enough bandwidth to work on that in the past quarter or so.

A

So uh this is slightly off the schedule.

D

So I think it's already already updated. There are just some looks like some formatting issues, but I think I think we can probably go go ahead and pin the api reviewers to take a look just to see if this is uh what they expected.

A

Yeah I have to keep what I was saying that you know uh this camp has been there since january or something right, yeah.

D

Yeah there was a yeah.

A

There was a good amount of time. I didn't get enough time to work on that, we're all busy. It's okay, no plane. Oh that's! Okay! I take the prime. If please.

A

All right, thank you, everyone and have a nice day. Thank you. Thank you. Bye-Bye bye, bye.

I