Istio Networking Working Group, 13 Sep 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio networking WG meeting - 2018-09-13

Description

- Controlling Policy Governance With Istio Authorization
- Istio-pod-network-controller
- Upgrading the Envoy binary

A

The networking community meeting is Tiana thirteen community meeting and we have two items today on the agenda. The first item is controlling policy governance with installation and I'm, not sure if Lehman is on the line or maybe should join us later in the room. And the second item is East. Your pod network controller and I saw Rafael joining so maybe we'll just switch the order and cover this in the beginning.

A

Meanwhile, could you please add your names to the attendees and, let's see if we can get started with the second item, so Rafael and Andrew I saw you joining? Are you able to to speak first.

B

C

Actually do change because you are in here we change the order. Is it okay to go? Second go first, I have another meeting, okay,.

C

Okay, thank you for that. Yeah yeah I need to go to another meeting for the second half. Thank you.

D

Let me present the proposal. Oh.

D

C

I, don't have them making you my 200, your present, oh sure,.

D

Yes, so I can start introduction, so this proposal is about using SEO oscillation for control. Who can edit of you is the policies so today.

D

So so it's new policies are currently expressed in the form of community status, and we currently use the communities are back talking, show who can edit our bill a particular and each new policy. So, for example, is the authentication policy and of oscillation policy or use the namespace as the scope to determine policy ownership, and so this model has the other advantage of. So it's fully utilized the pledge of what platform provider the kinetics provides all the control kinetise up and provides all can shown. On the other hand, it also has some limitations.

D

First, it does not applied to the resources like workload or hosts, which has it's not it's unrelated at four namespaces and it can cross multiple namespace of be a finer-grained object inside an in space and the other limitation is, it does not apply to a non communities platform. In that case, the namespace concept does not apply.

D

Yes, so I read, there was another proposal: traffic owner proposed ownership proposal which proposed to solve his problem by creating association between the e steel objects, like hosts our clothes with the namespace, and so this is just another alternative. We can do and I think it provides a more finesse rate here: okay, yeah, so here the goals we try to provide the access control for who can edit and for cambio an Eastern policy at different levels.

D

My coworker order level, then a host level name, space level or mesh level, and we want to have the oscillation work across the platform. So basically, this can work for Communities platform and the non-cognitive platform, and the access control should apply to all SEO policies and we want to provide our consistent policy configuration interface for communities and the easier users.

A

So, in the contest context of victory service, we had this problem of having separate ownership for, like different, let's say different, like paths belonging to different owners, but for the same host. Ok, so how could we solve this? Wait? Wait are back yeah actually.

D

Yeah, if you I, can talk about the different proposal. Oh you can see. Actually I include the pass. Okay.

A

So you have a concrete example: no excellent.

D

Yeah yeah I, don't know if we actually have the past example, but it's part of the parameter. Okay, great.

D

Yes, so so the proposal is to introduce at four new objects called an operator row and operator no Pandey. The operator row specifies the resources or sub resources operator is allowed for configure opium. The operator row. Banking assigns an operator wrote for a list of subjects.

D

So here's the definition of operator Rome our euro can include a list of rows. Each row is of commission, so each row can include a lot. The following entries: the API group, for example, networking dot instead of I/o or authentication top Easter dot IO. So it should all and Swiss is the adult IO, because we only talked about the Easter policies and.

D

Sauces other CRT CRT kind of ECL configure options: micro, virtual services, a virtual service gateway service entry.

D

If this entry is not our if resources is not specified its applied for all resources in the Aegean group and hosts, so we think also for our particular resource you can specify a hosts, for example, is applied for started in Opel calm and it can also other hosts. The same host entry can also express their own a space scope so, for example, NS one name: space can be expressed as start on: NS 1 dot, as we see doc custom, the local and it are the other.

D

The other entries IP addresses, so you can specify the IP address and also pass pass, can refer to the virtual service URL. What you're? Just imagine you? Okay, if.

E

It's we query. The IP address in this context is the service entry address that we use for for virtual IP addresses.

D

Ip address I believe several networking objects has IP address.

D

F

And I think you're probably watch one service also has one yeah.

E

Okay, all right just want to make sure that was.

D

Not very too familiar, we sit in there cooking objects and also pause you can use. You may also want to specify the pots and the verbs points so.

A

Okay, can you separate the have like separate policies for each port young.

D

Okay, yeah: that's why we have energy here and also verbs verbs, can include the catch list or create update delete. So it includes the read, read: operations and the write operations.

D

Yeah, so here the exactly meaning of the hosts like for virtual services, refers to a forced entry, ah yeah, so I I don't want to read all of this. Basically for ya, ya, difference the ugly definition there are different hosts entry. There.

D

And so this is an example of operator role, definition of networking academy, so so the the name of the ROI is the network and me, and so for networking a demeaning he's allowed to configure networking dot e still does I networking tell is the dot IO a key, a group for all the resources? Basically, I can even skip the resource count. Mm-Hmm and just for this particular hosts started example.com and.

D

This is another example which is security at me for NS one you're. Actually, the same function are here today is already provided for secret currencies because we already use in in space. So this example is using namespace, which is you can see. You can see the hosts entries the start out. Ns went on and easy, don't cross, don't open. So this is sort of security. Amy is allowed to configure security policies for and this one namespace that's what it means right.

A

So the name space itself is not a first class concept. It's just baked into the host, but if I, if I am to refer to the host by the DNS name, I would not have a namespace necessarily right like if my host is called booking for calm. Okay. How would I use the namespace situation.

D

That's cuz, there's.

A

No need space. Okay,.

E

If we're gonna, so I am not totally sure that we can do that, that method, to restrict namespaces in general and, like I, think we might need an ace field specifically if that is what we're trying to do. I also kind of worry it's a little bit of an abuse of the semantics. That's not like! That's not really! What host means there right, and it just so happens that the namespace as a concept is encoded in the DNS name, but I.

E

That's it that just that was the only part that stood out to me is a little fishy right. So.

A

It's not really.

D

True, this hostel.

A

Yeah exactly it's a bit overloaded semantics but I think yeah. Let's, let's hear the entire proposal because we had also like different alternative and then I guess the community will probably decide you know which one is the way to go.

F

A

Making better just you know at the options to it and all that right.

D

Yeah and also you can you can support the final grinder. Access can show like the host can specify particular service and.

A

The so they do, this policy apply Iraqi Kali like if I have a policy that says star and another person yeah. They.

D

A

When I'm coming next sorry, if I like.

D

Yeah- and we also want to a supported icon traffic smashing the suffix match in the host and class.

D

Yes, so awkward to row. Binding is actually very similar to the current. It's your service row binding or yeah. It's also similar to the community's role Andy. So so it has two parts. Why is the subjects its list as a list of subjects which you find through this row and the other part is the row ref, which refer to the low definition you can refer to the network enemy wrong, for example, yeah?

D

Potentially we can properly, instead of defining to different type of operator Roubini and the service row banding. We can just use a single row, Bundy object, which are quite opposed, because the definition is basically the same.

G

I actually think that the roll binding is closer to the kubernetes yeah.

C

G

Roll binding, because here we're talking when we're talking about about users and groups, we're talking about that specifically from the perspective of what the kubernetes api server understands right, yeah, yeah,.

D

So the user Evo will come from communities will be provided about communities that.

F

Means that people will have to have what about the user based authentication with kubernetes right, yes, is that a widespread thing that is always going to be there or is that gonna be I, mean.

G

It's basically table stakes if you're doing any kind of access control over sto the way that it is today right, yeah. Basically, we.

E

Just don't have an answer for identity in this context outside of kubernetes today. Does this your are back in general? Actually work outside of kubernetes no.

A

Thang, if there is no kubernetes, how would we apply this concept since some other, like platform VM.

D

Shining wind, that is, if your server will always be there, so so I think there's a.

E

Living galley not.

G

Given I think that server right like this is configured or access control over configuration. So so, when we're talking about subjects here, we're talking about subjects who who have access to change. Stos configuration not about subjects who have access to particular services that are sort of in the mesh.

A

Integrate with something as I don't know, Active Directory or like some other systems, for instance, would they be able to like enforce this? Would we be able to get you know some user account or.

H

Access control.

E

Yes, you're I'm, not on like it control, not kubernetes LexA. Nobody.

I

That this is kubernetes access, control for sto, just like Cloud Foundry today I believe they have some access control right. The.

F

Problem now is not about the access control problem is to have this at the point of config. In addition, so this is not like I mean if this is in Cabana does this is the lowest level of a stack when somebody is defining this through the CF API? Isn't so on we're not going to be able to catch this? It's all probably going to be coming through, like one galley user, always that's it and the same thing applies in any other CIC D context.

F

This will apply only in the expert user scenarios where people are actually directly interacting with kubernetes to write their entire configuration, and then people have, like you know, like everybody, has an account as Mike said, and then in that case yes great, but imagine something like fabricate or, like you know, any of those entire infrastructure that you actually overlay on top of kubernetes.

F

In those cases, the problem is still not solved because the problem just gets punted over to somebody else, because no CIC the systems are going to be the only users that are going to be generating these resources. Yeah well,.

E

So you can slow meant different user at different accounts for sei CD system to use. But my my Overton point was this: a should live enforcement at this should live in galle and how you get the user. Identities needs to be pluggable yeah.

D

E

That's the answer to decoupling it different.

D

So actually, if you look at the implementation, it is a plugin and.

F

D

There is a kubernetes api server there are we just don't, but if you know well cook, you know okay, so.

F

One position that I had is in stop having a laundry list of all the attributes that we can an operator can define like host IP addresses and so on. You scroll down towards the end of this document, I added a suggestion which I'm not actually gonna, show up but yeah.

F

So that's the session that I have switch it's more of like this makes it a little bit more generic and in the sense that it can actually apply to everything, are than just networking as well, where you define a template of some sort where the fields can actually have.

F

You know where the section is allowed or denied by this operator role, and that would probably make things a lot more scalable in the longer run, because, as we introduce new object, which has newer attributes and so on and so forth- and this also applies equally well to mixer configurations and or even the own are back configuration, so yeah Lee would simply have to run through this template, like you know where it actually looks at the individual fiends and then sees that's actually being overwritten by the incoming config.

F

And if that actually override is allowed or not, yeah.

D

So yeah, so this is a one possible approach and also I was also thinking of like a similarly I was adding to the service. You know for the exclusion semantics for the deny. What's you mentioned no.

F

Name is actually like a complete I, don't know portable struct or whatever it doesn't happen to call it. The point being that this operator role will no longer have any idea of whether you're applying this role, the role is applicable for IP addresses, hosts addresses and so on, because these are things that are very, very specific to the networking see IDs and you know if you go and take this and I like this too.

B

F

If you want applies for mixer stuff, then there will be a whole bunch of other things that come in and we'll just have way too many attributes it just becomes unusable. You study, you just create this as a very generic template, something that very supplier blob like that. Like the way we do validation today, right so like pilot, has a validation file which has all the validation, stuff. Miksa probably has something similar. So we get to write the stuff and the semantics of that proto thing is and a fully enclosed within pilot mixer.

F

You know it's a little and so on before they taking the blob and then they take in the template and then say this is allowed or denied, but said so then the system is enforcing. This will become much more simpler and easier to maintain in the longer run. Okay, so.

A

Okay, would we be able to model something like for this role, allow at all the permissions, but deny permission to edit this particular resource.

D

A

D

F

But I'm, basically, the we'll also encompass a traffic ownership proposal where we say that hosts started, phu kham cannot be modified. This is always read-only, but people can define something under started. Combat like you know they can only add additional HTTP rules or with these parts and so on, and at the same time you know you can also define that nobody can define. Something like hosts are calm. You can actually do an outright deny, which would have actually allow this. It should encompass the use cases.

F

The traffic ownership proposal asks like who can capture wild cards and who cannot capture wild cards, and if you have this this way, then the nicer properties that we can do some form of inheritance in the destination rule where somebody can define a global destination rule which says MPLS everywhere for all clients, and we can have the inheritance semantics where, where we can actually set up.

F

This are back who that says: nobody can overwrite that the TLS field, a destination rule or internal hosts, and then then we would be guaranteed that, like every destination rules, the user supplies will not have the TLS field. At all, you can always inherit directly from the global thing today. The problem is that they can't do this, because inheritance is not defined because we can have user can define a local destination to run a global one, and things are just so I I.

E

Do not want to couple the notion of access control with the notion of default configuration. There should be.

F

Coupling it what I'm saying is: if we have this in place, then we would know that any destination.

B

F

Not have a terrace field set, which means there is no conflict in merging it's a parent rule and that becomes an easier one and the same thing can be applied applied in in other scenarios, where somebody wants to define a global rewrite across all virtual services. Today we can't implement that, because, inheriting a virtual service, like from Starcom to start out to outcome to start out, bar.com is complicated. What happens if there are conflicts in the same field and so on?

F

But with this we can actually go fine again and say: you know what, if you only the route can actually do that Global rewrite and everybody else has to simply inherit that phim. Nothing else can actually be done, and so.

B

This can actually.

F

Become it's it's more than just like. Allow ordinary code actual become an inherit as well. Waitjust yeah.

D

So, by the way are the templates is it can be reused or it's just defined here. The are defined for every policy, so.

F

Every resource type makes it so for virtual service. You have I mean this. Template is take the same proto, that virtual service has, we probably go and add a that are back, feel or whatever it is to it all. The places that could potentially have inheritance or well there's a hand.

E

Railing but you get the ideas yeah so like we can use field masks to achieve the same thing. That's a horrible API, but it's the same semantics as one way to do that. So you could just the the binding itself for this rule itself could just be a list of strings or a list of field masks that target the fields of the virtual service or you could the lecture and suggest to do like a CR D per object.

A

E

For resources so.

A

If this were a very.

J

German, instead of having hosts.

A

And IP addresses and all that separate, then you would have a field like the ability to restrict modification of a film that the user would define and we could we could in this case we could disable access to like TLS settings right, but.

F

A

Would not be baked into the implementation of the policy to be something that the users you know, configures.

F

A

F

As far as saying that, Albany is operated role and operator role binding and then in every CRD that we currently have in sto and in places where that could actually be overridden and whatever it is. We asked to add on in feed in every I. Don't know the sub struct that say it's like you know our back more disallow deny it's an enum that we actually have to add so that when people, whatever template that people defined here would actually be. You know that I don't want to stock.

E

Up all our API definitions with with our back stuff, like those those are two independent concerns, the objects themselves and the policies over the objects like if we want to do this kind of generic stuff over top I think we can totally do that, but we can do it in a way that doesn't require us to change the actual SEO API objects. I, don't.

A

Think a man changing the API, maybe you were referring to the implementation she around right because we don't.

F

A

F

Yeah IIIi had meant thought of like changing the API, where we actually add additional are back fields and then, which will be like not used when people define and stuff but they'll be only used in this template mode. But Zack is right. We can do it in other ways as well. You don't have to couple them but decide.

K

F

Across other sto resources in the right.

K

A

Like there would be too much complexity for the you know, casual user, if they have to think about, do I want our back or not. What is this our back right? So we want to start simple and add features on topics you know just like we had security or telemetry or our back. You know fine-grained control policies, so yeah.

F

The type one yeah yeah.

D

Okay, let me continue on what we left, so you can enable this operator on solution a particular namespace or you can enable it for other supporting resources basically included, although is do communities so so this is actually the same semantics as what crunch is the Americans we have mode which, which includes the arm of always encroaching on with X Croce, so always X.

D

Coaching basically has the white race to de namespaces, our resources, you know and always excursion Besame blacklist, the resources that shall not be authorized us through the history of oscillation, and here is a example, so we use them out always encroaching, and the inclusion based include networking policies and I also added an operator row and operator row finding itself in the incredulous, because I want to support the Commission delegation which which I will talk about next.

D

So so for hosts and the IP addresses they have the natural hierarchy. You know. Actually, entry, oh and you're already mentioned is oh yeah, who is one of the yeah, so we need it yeah. So basically we can use the kubernetes other to bootstrap to create the mesh level, natural natural enemy or security on me.

D

So so, for example, you can create this operator role, which is called a global network enemy which are quite holding in high mesh, and this should be a cricketer through kubernetes, now back control that kinetise are back and after that the global network Academy can interoperate does stop enemy. For example, they can create the network Academy for started example.com.

D

So you can just include the hosts entry here and you can so for the example.com Network Academy can further create another another operator role, which is called Network Academy, for example finance. So it's only applies to all started, financed on example.com and so similar to IP addresses, and so the the general rule is again. It is allowed to create or edit an operator, row or awkward opening, even only if he has other permissions that in turn, if he has all the Commission's that are in the comic regular Rowan romantic.

D

You so so here I was saying in the initial offering maybe we can just let come necklace up extra control, although creating or editing for operator ruin, operation, open, D and then let other the hierarchical suppose later.

D

So, how does this is the or operator operator row and row? Binding interact with communities are back so first, if you are further resources that you want, who wanted to be controlled by is the oscillation.

D

You should exclude this from communities. Other policy, you don't want to apply for controls because it will just be confusing and it may conflict and also for T for the operator row and aqua clear opening. If we don't, if the permission delegation is not supported or used, the other operator operator appending, sherek insured by communities attack.

D

If we support permission delegation, then we should use kubernetes. Our bags for bootstrap basically created a mesh level aqua to know and Robyn D, and after that the control should be switched. For instance, addition, so you use the you use the operator run or it'll open itself through control who can create additional room?

D

Yes, so here is the proposed implementation.

D

So we want who implements this operator of oscillation entry inside carry and the import for the Osaki trim will be the ocular row and operate a row pending and the other is the public objects which will also control, and we want to have two plugins. Where is the oscillation webhook? The other is admission control hook.

D

The reason we need to is because our mission can show webhook actually does not support the read operation. So all the operation will be authorized as through oscillation well cook and the web poker. Ascender are subject: actors access the radio request to the to the oscillation engine inside any and for for the write requests there will be additional admission admission, control, review request, which will contain the rupee to a modified object which is not included in the oscillation well cook, request.

A

So also I wanna clarify, if you think so for this this to work. We need golly right so because, obviously the plugin is in garlic and what is the current status for far back in general? So let's say we have like some basic are back functioning and we need to add the new operator operators and operator roles like the associate that you described in the document, but other than that there is already some. Is there already a beginning early that foreign.

D

Samurai carb are functionally other hands up a face: the supported a long time, Rafael solution, it's actually a different module from this one. Ok,.

A

So that would be of brand new thing in golly, okay, yeah yeah,.

D

It's a yeah, so the current epoch works on the ovoid calm, its authorized, the wrong kind requests, but what we propose the here is to add a config file, a solution, ok and that module need to be prominent. Organi.

L

So they may I have a quick question. So does this also apply to the authentication policy? Also the example on the networking yeah.

D

L

D

Indication policy- currently we use to the communities, are back to apply namespace copter access can show. But of course, if you want final, green or other levels you can, you can is the oscillation as well yeah. It's.

L

Applied to oscillation.

D

For education policy so.

L

It does apply I'm trying to understand for my user perspective. Would they be using operator row when they see you know the API version? Is it still dial in their config, then they would be thinking about using operator row and operate a row binding to control to control the also in authorization. Oh, do we I mean? How does the user know when to use Kuban any are back with israel authorization? I think that that's not super clear.

L

D

So I think the guideline we should give this to a user is if the current namespace based or sedation is enough, which is provided by communities are back, is enough. Then they should just stay with the community topic, but if, if it's not, if they want to find a green like her, they they actually want namespace level access, control or if they are running, are now communities platform. They don't naturally have the namespace or if, for example, for the networking policy, they have hosts the concept which namespace cannot restrict.

D

Then they should use in still operator row and offerings for open, D. Okay,.

M

If it's probably useful to include that guidance in the dock, yeah.

L

Okay, I think that would.

A

Be really good yeah are there any other questions on exact.

F

A

We'll have to go to the next topic. Yes, I just.

I

Wanna make one note on the implementation that relies on the authorization web book that will require restarting the API server in passing the that what book configuration to to it. So you tie the lifecycle of galley to the life cycle, the API server, which I don't think we want to do well.

E

I mean that sounds like that's actually just broken at upstream kubernetes I mean like I, understand why they might want to do that. To tie the life cycle of their API server, to whatever is doing up authorization I think.

I

People want to fix it, it doesn't work today, don't point. So if we go up and do this, it's it's not going to work practically.

E

Authorization is only required for read restriction right, yeah right. We.

D

E

D

Right there, you just do a Mission Control. If you all care about controlling who can read there, yeah.

E

So it sounds like we can like go file issues upstream in kubernetes, maybe get some of our friends around the corner from you all to help and then ship like read, there's a second pace.

E

Yes, do the admission yeah should ship the other access first and then we can get that later.

A

Any other question burning question that cannot be answered offline in the document.

A

Okay, so I think, like I, think we got a very good overview like pretty detailed in fact, and the world probably. We need to leave some time to people to like evaluate the proposal. Also like look at the other, the other, like the other proposal and we'll probably make a decision as a community and at some point like.

A

If this is you know the way we will solve the problems with the virtual services, Mikey and I I think people will get into more detail into commenting on the you know the API here so yeah yeah yeah.

D

Yeah please the comment on the talk. If you have for the comments for there feedbacks.

A

Okay, green okay! Thank you very much. Thank you. So I think we can move to the second item, which is the SEO for network controller, I, hope, profile and Android that you are still here. Yes,.

B

A

Great, thank you so of please feel free to present I know like like we reviewed last time you had some slides, so I'm yeah I'll.

B

Do I take ownership of this screen? Oh yeah I think.

A

You just like in the meeting you choose, please I can stop presenting, also I'm, not sure. If that will help I think.

B

A

Or seeing something that must be you there, yeah yeah yeah.

B

Yeah, so if you remember I think it was more than a month ago, I presented this idea that Andrea, Andrea and I have been preserved and been developing.

B

What we are trying to do is to increase the security of issue by removing the need of running a privileged container, and the container that needs to be privileged today is the innit container, the one that sets up the IP table rules.

B

So I tell me if you want me to go again through the technical description of this proposal. My intention today was give you a status update of where we are and then talk about a possible PR to the project. To the main is your repo okay.

A

I think that sounds good. Does anybody need a short overview?

A

Okay, yeah I think we can probably skip the overview because people still remember from the last meeting and we only have like 20 minutes right.

B

So last time we had these open issues and I want to tell you how we address them so for the first one was: how do we know that a pod should belong to the mesh and therefore the demonstrate whose which sets up the IP tables should actually act on that pod and and what we decided to do was to add an annotation to the pod that works in a similar way to the annotation.

B

That says, if a pod should be injected with that mission controller, so, unfortunately, it cannot be the same annotation, because we have slightly different logic, but it's so it's an additional annotation that works the same way and then the second was it's, it's a docker will be add we need to. We need to do.

F

B

Interaction with the power name space, so that piece is is dock is container and time independent and the last time we only have the talker implementation working, and so we we now introduced a cryo implementation.

B

It should it should be CRI compliant, but I haven't tested with any other CRI implementations except cryo, so I can only say right now. The project works with both darker and cryo as the container run time and questions so CRI.

A

That's container runtime interface, so that would be the like the layer that obstructs from the actual container runtime right. So you should be able to use docker and rocket and other like runtimes, with CRI yeah.

B

That's that's true. Although for docker we have we kept the current docker specific implementation. Oh I.

A

B

Yeah, that's better right and then the third point is, we didn't have last time an explicit synchronization between the pods and the container starting in the pod and the Taman set, which sets up the the IP table rules. So we introduced an inny container that it's not privileged, but it waits for the ford IPTV rules to be set up so right now we have full full synchronization.

B

A

Where is the need container? It's in the controller pod in.

B

Container is in the pod that will join the mesh.

A

In every podium and there's still an init container, there.

B

Is shalini containers just not privileged and it's ejected? You know it's transparently through through day through the normal injection. You know mechanism we just we just.

F

B

We just change the injection config map to to inject our container instead of the current one and.

A

How do you wait for the iptables? How do you know they've been configured? We.

B

Annotate the pod, when the pot that needs to join the mesh with a with an annotation once it's completely initialized and any container reads that annotation as a as a config map mounted to the pod.

A

But so so the pod can be also annotated by some other actors. Right, if it's not like protected, anybody can add an annotation.

B

M

I mean you could even restrict the existence of the inner container, which is going to trigger the the network initialization as the annotation or, if it's alright, your logically, what that in a container is doing is saying, join the mesh right.

B

What any container.

B

Yeah sure I could do that. I see what you mean. The any container could could annotate the pod. The problem is, we didn't want to give any privilege to the inner container, so instead, if it needs to be able to annotate the body, it now needs a grant or on the kubernetes api. We didn't want to do that, but.

A

I thought to use the unit container to check if the pod has been annotated by the.

M

A

So the purpose is to see that the IP tables have been configured by some other entity, which is the post controller route, runs.

F

A

Of the application pod right so they need, they need a way to signal that that this has been done. Great.

B

Let me see if this will I'm.

A

Not sure annotation is like.

M

Hey it's the mesh and the demon says you're at it or you're, not I.

B

Do well, how do you do this? A demon add me to the mesh without granting any permission to the any container by.

M

Making anything I called me.

E

M

E

M

Right so let's say that the demons running on the note it's listening on a domain soccer or a pork, that's no local right! Then a container boots. It.

A

M

A

It's it's, it's probably less trivial implementation than using the annotation like checking for an annotation, but I think it's also bit safer because you can have anybody like any bad doctor, annotating the pod and then just like bypassing the entire IP table stuff right.

M

So I mean obviously the net container requires no privilege the contents of that API right can be evolved over time to support more complex use cases around initialization, which might be useful all right. So really. The goal here is also you to push the complexity, the network initialization down into something that's already privileged yeah right by the demon so so, and then container weight.

A

Yeah Oricon error.

M

A

And they find that pods nettle controller in the future me to other stuff than just configure. Ip tables right like if need be, yeah.

M

The same mechanism would be used.

A

M

But the pod doesn't need to know that right.

A

Okay, the pod means know that.

M

B

All right I like the direction but I'm just trying to think out. So these are the actors right you're, saying this: this mash pod, as in any container that calls the network controller.

B

It would have to call the network controller on the same node. Yes, but how do we do that in in in kubernetes right.

M

We can have the main socket right. Some examples of other people having demon sets they're being called by the node by pods. What.

A

What privileges do you need actually for this.

M

Number call: okay.

A

M

The only thing you have to worry about is the security of it right kind of be spoofed or not. Well, you can tell us for that right, simple deal us that can be simple to us right. We already call like the Citadel agent on the node to do stuff right to get certs, and we normally have to wait for those certs to actually start the network anyway right. So we're already doing this.

A

Yeah, except that's, that's different.

M

But it's the same: synchronization properties, uh-huh.

A

Is that part implemented already.

M

Know it currently races because there's no way to order it in a container with actual pod, there's an open issue in kubernetes. What.

A

Do you mean because it's all these little.

M

Starter clients: it runs before right right, but.

A

I that you don't know about the node agent like when it starts no.

M

You know that the demon sets are supposed to be guaranteed to be there on the node right. The node is not scheduled until the demons that is running unless you're, adding the demon set after the fact and.

F

M

Race, okay, but that we have an issue with racing with the init container right now, but.

A

Against what is it tracing the.

M

Sorry, not the the Envoy pod races, because it's still waiting for network config file, ATS X, yes and but the actual plot itself isn't waiting.

A

M

That's for readiness, it sure that.

A

It shouldn't like it shouldn't what shouldn't be ready, honest with signatures.

F

A

And we can wait for network configure the config and see it's ready.

M

But anyway, something we're obviously doing it once these things today, and this might actually help us rationalize some of that, a Keith. What about it.

M

B

Implementation, not still I, don't think I've ever seen out how this can work so I know.

M

For example, that in calico they call a node agent, a daemon set notation over the main socket to do orbit checks, and so the file path of the demon, that's listening on the main socket, is mounted into the container yeah. That's that's the.

B

Way another you know that that's it's still a privilege action to mount our stuff. You.

E

Can do that a bit sample layer before it's deployed I can do it you can do it is like I start afraid you don't have to do that at deployment. Time like you can do that ahead of time in the template, for example, so that that's so that there's no privileged action, that's occurring at runtime. If.

I

You're reliant on the injection, yes method, to set up everything properly. So if you're gonna discuss trust.

E

B

I

Do it at a level.

B

At least in the opposite model mounting our path, mount is a production that normally you don't have I think what you're saying is is mounting a socket at the main socket, which would be basically a spaz Mountain, correct, yeah.

M

B

So that yeah that's another yeah and I, think it's the same in the pod security context, model that is coming up or kubernetes.

E

If I that's a privilege action, if the mount is explicitly listed in the pod correct so.

B

The way the way this work, this works is, you will be, you can't constrain the user to what type of mount they can do and they they will all you know the least privileged action is to mount a persistent volume usually ask mount, are considered very highly privileged type of months, good cause. The idea is to protect the.

M

Nodes are protected right, they're, slightly easier to spoof at the network level Bob, we can use an array key. We know the node IP right.

I

M

A sufficient solution and doesn't require any privilege and for people who wanted to be to think about their effects a little differently. They could obviously configure their own operational behavior to mount the domain socket. Instead,.

A

So the file month was an alternative to the domain socket.

M

Dizzi file yeah.

A

B

So, yes, the the IP of the node can be injected as a normal configuration to the card, which would then set send a request to the.

A

Downwards API is I'm.

B

Just checking in terms of privileged provision.

A

B

That a support makes a request. Today is about no controller and then was sorry network controller and sending some identity right. It needs to. You need to tell that it is, and then from there we can. We can move on basically.

M

B

E

By the way, can you look this stock in the the working group right now? The only thing that's linked is they get every vote? Yeah.

B

Their most presentation attack yes, and this. This is an article that we are writing. It should be out in in a couple of days, but sure I can I can link this document. Just not thanks, not sure and.

A

Like it looks like we're, gonna adopt this solution right. That's why we're discussing about so do you need help are like? Would you need people to review the PR because it still do you need? How are.

B

You ask if somebody can volunteer to give us a tour of the get the git repo and to understand how we should merge the code.

B

That's what I meant.

M

I think there's obviously question about point. We now are going to have two solutions which one is going to be default. How do people choose which one we're coming commingle? You.

A

Mean the dissolution with the one being what we have right now start to scroll and this being right, yeah right, so that would be like baptized into our like health deployment done right.

M

So, theoretically, this still the open question about notice right at some point. Well, what the hell are we gonna do with this solution? Doesn't work for notice.

M

It runs on nodes that you can't are totally virtualized and don't necessarily have demons. That's remain on them unless you're gonna have the notionally virtual daemon, which is kind of like a thing that exists, a logical node, that's all unresolved right, but it's an open dialog in the cabeza community, I sure shipped a product rate called me KS with I. Don't know it was it either. It's either infinitely scalable load or it's something else, depending on whom you look at it. But.

A

That both has large implications also on our security story with the node agent yeah.

M

Anything right now, depending a node agent,.

J

M

Even the vendor has to pretend like the node is infinite and it just runs all the containers and then that has an implication for the vendors to figure heifer when the demons mm-hmm.

A

Right but I think for at least for until that becomes a reality. We can probably proceed with with. This is an option right then.

M

Right right and then they don't know, the thing is not everybody necessarily one actually wants to use iptables capture right. They might actually be willing to write the app to just talk to the proxy port and.

L

Then they don't need anything right.

M

Between the third flavor.

B

All right so I contact you offline through the mailing list to see if we can set up a tour of the git repo and start talking about how we're going to do the marriage a meantime. We were going to design on based on the feedback around this different type of synchronization. So a.

M

Question I have for other folks, who've been maintaining this stuff is. How do we think this solution compares with the existing solution for a proxy upgrade.

E

I mean well shirin, go ahead. Oh.

F

For upgrading an existing data pane right.

M

So let's say in theory: let's say we don't have to do rolling pod updates, but we can do roll envoy updates, depending only on where the old ways are running. We.

E

Saw a pilot agent in the Envoy container to do that could do binary upgrades right. Yes, we still love that. So that seems more desirable to me. It feels like you wouldn't wanna I mean it depends. What your security model is. I. Guess right, do you trust the daemon to the point, the new binary? It's everything on the node.

E

If you're trusting the demons to deploy.

M

The binary or pods and implicitly you do that right, yeah.

A

It's gonna be very hard to separate the privileges. Then, if you give so much privilege to pilot the agent than any it can download, you know the between what.

M

Is pilot at this point.

A

In my opinion, if these are necessary and we should get rid of it as soon as we have STS right.

M

A

M

A

That I, don't like the idea of adding more functionality into pilot agent.

M

Right, so really, what we now have is this injection this in a container that says: hey demon, inject pilot with the right properties right into the network path for this thing, which basically subsumes the role of Pyla agent, alright, but then we're assuming. We fix the other problems right, acting like there's, health checking, a readiness or other.

F

M

Have to deal with does that normally also mean that there's also a way to say: hey demon I want you to upgrade all the envoys on the node or the on these envoys and does not actually make it easier. So.

A

By demon demon the same the.

M

Node agent, like.

A

This pod Network, it's called thought network controllers.

F

A

This guy, not the other one, the node agent for security right. This will be make more sense to give okay Arduino.

F

But I mean it's it economy, in-place upgrade I, don't thinks all right. It's probably gonna be something which requires destroying the board and we're well having doing a whole kubernetes rolling update for the pod. Remember.

M

We can always says I can do HA update for at least a passive process. You could maybe do that. Well,.

F

How do you push that container into an existing pod and binary you'll, throw into the existing or and that's the bigger problem is like you can create a special mount within the pod such that you can always like. You know, mount a second version of an Y, and then the demon can actually also exact into that container.

M

You know we don't. We.

F

M

Have to like push envoy into the pot, except in the C group, sense right.

M

F

If I'm at a higher level of.

M

Privilege I should be able to start a process and maybe part of the C group.

F

By default, in all the hosted platforms, they're like you can only meant set that would do all of this stuff. It's like well, I, don't know if it works, then great, but I mean an easier approach, would actually need to have a amount, a worldly mode or something about sort from where envoys actually started up, such that the demon can decide to stick in another NY binary in that same mount and then literally exactly every port and hot restart that envoy and the two things would automatically transfer and it's just zero downtime for a pod.

F

You don't even have to do the kubernetes rolling updates, and you know that would also just work like right.

M

Those are kind of the same thing, I.

E

Mean so do we still have any real use cases, though, for for like binary, reload I know, we have hypothetical use cases like binary.

M

Upgrade you know just the mechanics of doing actual upgrade yeah.

E

I mean yeah and I do think that there are some attractive properties about doing that at the node as far as like not having to deliver the container binary to every single pod and delivering it once to the node and well I, guess actually that already happens automatically yeah.

M

The other thing is obviously the note agent doesn't technically have to run on boy in the pod, but then that changes the security right.

E

Change to change the security.

M

Model and it changes the resource attribution model well,.

E

And you basically go back to I mean yeah I I, think that.

M

E

Breaks the identity model in sto as well I, don't think we can really terribly do that. We have no need a large parts of fine art. You have.

M

E

For the handshake frame.

M

Except the pod we'll have to do the the cert validation during the TLS handshake, whether those be pieces with the best side.

F

A

Here, talking about middle proxy, if we don't enjoy.

M

That it's it's worth, maybe a plication. The network is a part of sorry. Illinois is no longer than the trust Amina, the pod. You could slug Lee. So basically the networking would be a-okay to it completely, including the existence of a sidecar.

M

The only thing and the question is: what's the security model, you can at least have a situation where using an outer band handshake mechanism, it's the key material is only in the pod and therefore the only handshake can only be granted if the pod basically allows it, but something running at the node levels he's all the traffic in the clear after the handshake. So how do you.

F

Do that because I mean especially the inbound connections like the poor, you have to have something running in the pod, presumably some version of an envoy, that's trimmed down that terminates a TLS. No, we already have a mechanism for doing a verb and cheerless handshakes.

M

By an eight-yard cowboy already supports it, so basically you get that hello and you make it the main basic. You call another thing and say how.

F

That lines, lines, authentication thing: yeah, okay, Oh, so.

E

What's private level well,.

A

E

Are serve all addresses.

A

Consumption because, like all on a node you'd, have just a single employer.

F

So you still have.

A

One in every port, but then you.

F

Say it's not running in.

A

The pot- it's not the sidecar, so it's your note. No.

M

No, it's a one envoy for part, but the part is not. The Envoy is not running in the car. Oh.

A

So Benny, why don't we do the full middle proxy? Because we can because the same like.

F

A

Envoy, if it's just able to forward you know the TLS, and it's still the actual note that has its the actual product has the certificate right then now it doesn't need to be for.

F

A

F

If we are going to run an envoy for every part, but inside the demons thing, then then yes, I, guess the existing model will still remain as it is provided.

B

F

We just have to have a way by which, as you said, the client authentication stuff needs to be done, which we can actually go back to the power and do the stuff and come back. Yeah.

H

F

No but I mean yeah I think the advantage is that you probably have, like you, decouple the pod from any unavoidable stuff that we both do. But we it's all aggregated into one giant demon search for which people allocate resources.

M

Problem is the.

F

Resource attribution, Oh.

E

Resource attribution, the trust model becomes a little bit harder to understand, like a lining in the pod works very nicely in kubernetes right.

M

Semantics, like.

M

Solutions that are very demon say having the.

F

A

Said even if it has like ideally I would like to have a sort of a multi-tenant and we're running there.

K

I know a whole, but the whole plan.

E

Is it weighed all that complexity right like the sidecar should be lightweight in a not so console.

M

A lot of effort.

E

Into making this overlay wait right, but that's but like that's the whole premise of the pattern.

G

F

Don't have any soft multi-tenancy, it's not hard multi-tenancy. We need like admission control the source, isolation and so on within that single arm way. Yeah yeah.

E

F

More harder problem, the solar, especially when people started asking for a harder, Moulton dancing model, whereas within a single part, it's just cattle, and then you allocate Motorsports for the poor and less useful for the poor and.

A

All right so, let's say even if you have more m voice training as a diamond set instead of running inside car. To me this is still a better model than the sidecar model like we couldn't implement this earlier, because I think some of the constructor missing in envoy right. But it's about how you control that diamond, set pull of n voice right. You can apply different policies so.

E

The whole point of the security.

K

E

Is that you get to use a single tenant proxy? That's vastly simpler and so in theory can be faster and use less resources, because you don't have to worry about heart resource isolation. You don't have to worry about all of these things. You get better cache locality. There's all the right like it's a it's a it's, a large fundamental shift in the model to trying to promote yeah.

M

I, don't think we want to move to a multi-tenant envoy it would. It would make the configuration story very difficult, pretty clear on policy yep, that's not to say that you couldn't have run an envoy tenant.

E

So that's reasonable, right, and so we'll be out of a pod I think is may is, may be reasonable, depending on what your security model ISM, but what you want to optimize for there, but certain I think one per node is not something that we really ever wanted to do. Okay,.

A

So that would be one lieutenant or no, then yes, one.

E

A

M

One poor pod, but.

F

E

Her identity well, so you so run into resource isolation issues there if it's right well,.

F

I can have multiple.

E

Pods of the same identity schedule together sure but well.

M

Why would there's no there's? No, our security privilege problem no.

E

But there is just like a so, for example, like I'm police, already, a bottleneck for some applications. Now I have one between two applications where I should have to can I don't easily dynamically assign no.

M

Identities, our scheduling time.

E

Yeah I mean so when per identity is the minimum that we could do and still preserve the current semantics around our security model. Right so, let's say, for example, you with a circle.

M

Solution that ended up killing lots and lots of duplicate pawns for the center identity yeah. They.

E

M

Them a concern, yep dad I think is a very reasonable use case yeah, but we still have the resource attribution problem.

M

When Envoy can see in the CPU and memory which, how do you assign it back to the pond, so you can account for it? Oh.

A

Okay, so four billion basically.

K

Horizontal scaling, you're.

A

E

It just seems like we're doing a lot to try and defeat the kubernetes model for benefits that maybe they aren't necessarily clear to me like no, no I mean I'm, not so.

E

F

Whatever we do, we still would have to support the sidecar style model with the consolation generation for other platforms. Yeah.

I

F

They cannot, let's say for example, share this. Can I have a thing and without CF over there is the plain old vm's and so on. It will shall be like a sidecar per a binary of BM I, guess in the VM sidecar per VM, but still and the way we generate configurations cheated, treating it as a pod and so here's a whole bunch of configuration. So if anything, we do ends up changing that. Then we have this duality problem that we also have to generate this envoy.

F

You know for identity, with all the configuration and the same time. Oh, this is actually running on a I, don't know BM kind of a thing. Then we also have the generate conversation in the current way, which is the full-blown configuration I. We do right.

A

But for VM right now, I don't think you can run two services in the VM with the current model right so.

F

Actually, this this.

A

This demon said that model would help the use case. The.

F

Packaging becomes harder because actually, dementia, at least you can just spin up more containers and so on, I, don't know, and the VM thing you would have to have something else. That is like looking at the number of services that are assigned to that VM workload and then based on the dynamically spend some more on voice and we have to create whatever that's necessary to like. You know, recycle down my manager, lifecycle, Apollo, the on voice and so on.

F

These are vastly easier within that single uber within a pod of its in a container monitoring, because it just takes care of you know, booting, this container of stuff fails it in a VM. We have to rewrite the whole thing. We have to do it on our own, which it's just work on the ground, but.

A

But I don't think in a VM. You would need more than one right, because it's like a VM is still a single tenant. If we go with this idea of like 1m void per note per tenant, then your VM would still have one envoy, but that then we would be able to solve like more than one service.

A

M

Services is fine. That's on your problem right.

A

Now because it's the same idea that you have like the sidecar, so you cannot have more than one service.

M

A service you mean scheduled workload or doing something else, I mean.

A

Like a binary running in a VM implementing a service, so right.

M

K

A

Two of these using the shared envoy, yeah imitation.

F

Of the mesh extraction, but that's the problem with the policy. That's a column is everything else, but not with the actual running. It's a fact that makes a Condit. You don't mind whether it's going from someone or service to right, and that is where the multiple service Forum comes. It's a more inherent column and in our you know, the actual policy design and not with the fact that we cannot run multiple language because.

M

It is more binary there yeah, they have so, let's not disjoint ports unless they're super magical, which means they can still be addressed and mixer can still distinguish them nor I. You know whether we do a good job with that or not. Today, that's separate, but thirty. We code.

A

We also said I'd be tables in the VM, so the IP tables are said to forward like winters of the traffic and send it to a single service. It's using it's not if we.

M

Have to inbound ports to declare them bound ports in a pod right we have to forward to in oh, no, there could be those ports can be listened on by two distinct binaries running anivia right or a pod yeah.

A

Actually, maybe yeah, maybe that still works I, don't know what the limitations are, but I might be.

E

A little bit more complicated but yeah yeah.

F

Anyway, so I think I took namespace at all to be an issue because journey has containers. The the demon said. The whole thing is gonna, be one network namespace and that might actually create an issue, because you know this one white listening on the same 15, thousands, it's not gonna work. So we have to change that thing. Then you know if every hard-coded port, that we have 15,000 series port we'd have to have multiple sets of those 15,000 16,000 14,000.

M

So so, coming back to the more a more basic part, if we have an in a container that basically calls the demon, then says: hey inject me that at least gives us flexibility to control how the injection occurs without change in that contract yep, which is right. If we want to support more than one of these models of these injection models, then that seems advantageous in the long run. Yeah.

E

And well and I picked out, aligns with some of the other like identity issuance problems and that kind of stuff maybe can be simplified with that kind of model. I know that's roughly what the note agent does today right, but yeah I Jiri earlier point about, like one abstraction to solve those that set of startup problems. I think it would be nice.

M

That's gonna observe some thought, but it seems like a reasonable approach. Yeah.

M

It also makes it look a little less scary to people when we inject stuff- yes right, because all they see is these. Do an injection saying like this tiny little binary, then all that does is talk to the daemon which is under up right. So now yours is a stronger set perceptual separation between the service owner and the network operator, which is probably not a bad thing.

A

Okay, we're largely over time, but it's a very interesting discussion so.

F

Let's start with this and take baby steps and make sure that the you know the privileged thing is removed, and then, let's see how far that takes us and that it still give us the mean size, is that right, I think.

M

We need to write a dog because we're gonna have to get this reviewed by some other folks, I'd like to run this by some folks in Cabrini's networking world and actually just in kubernetes world in general, because this reliance on a daemon is interesting and what's gonna happen in notice, I'd like to get some feedback on that. Well, he has long term tenable or whether there are other solutions out there to this permissioning problem.

E

And the theme didn't style, the team that style helped us at all in non cube environments, I'm, trying to think having a daemon that does that might be actually that might be kind of convenient.

M

Certainly in managed VM environments right there is a most expose, a metadata service where you can ask it for things: I don't know, even if that would be something that VMware would ever do right, but the hypervisor provide an API where you can say: I want the mesh network.

F

Well, the mesh network is what I mean: that's a l7 network return I'm laying out for that purpose. Well,.

E

Just saying you know like inject me, aside card, basically all.

M

Right yep right, so we came.

E

M

World, where you can't rewrite VMs or doing like injection like you, can do in kubernetes, you still want to be able to control how you turn on the feature and where that feature not belongs in the hypervisor like hypervisor, is the node agent effectively yeah.

F

But there are other mechanisms like instead of running that thing on a VM I mean you can just still learn it as a co-located, VM and the traffic can actually be, like you know, interceptor and outlet through this VM open this wish thing or, but if he allows that kind of a stuff, so you do not have to you have to run run the VM and say injector. Well, yes to your point. Yes, you can have and.

M

It's not so again, like you wouldn't make they inject sidecar. We would say turn on the mesh yeah. That's what you would say.

F

Okay, and that would effectively Oh enable.

E

D, it's like in cubes. That would mean in check the sidecar pod right, but in BM that could mean start the sidecar process, or it could mean some other thing right. If you were gonna, try and hide it away at the node level, for example, and another.

M

Concern that I have that we have to think about is cubelet health checking or, if there's no declaration in the pod, for the thing to health check, and you want to health check. The Envoy I always keep looking to health check. It.

K

A

K

A

K

A

F

But be country we can container in our dream and said so we well I. Guess that brings a lifecycle column to us. Then we have to make sure this thing is not down, and then we have to make sure this thing is backup, which is a bigger problem, because a nicer thing about coherent is that it takes care of a whole lifecycle thing otherwise you're getting into this really really painful thing. This whole bunch of, like you, know folks and exits and managing the zombie processes and how that's a really painful thing.

F

Actually, then, you start looking at like where this thing has died or not. If it is like not dead on, and then you have to make sure that you know if that thing is actually working, something make sure it's just not you know problematic, and when that dies you have to reap up reap all the resources.

F

It is actually very painful, and the nicest thing about the Cubana does abstraction is that even if it's a container that will take care of like cleaning up that stuff and if the container dies, it'll restart the whole container and which, if you do the daemon said well, we don't declare as a container, then we have been hurt entire problem yeah, but.

G

What we apportion.

F

From the pile I mean, this was part of the pilot agent thing in the beginning it took us a while to stabilize in just one envoy- and this was from the old amalgamated days, and it took us a while to stabilize back and forth with like whole bunch of retries back house and add on this, and it's probably gonna get more compounded with. We have multiple envoys are dynamically being started. Adam I think.

A

The demon set also starts containers right, the demon set is just the thing to save right run it on run, one.

M

No, we got to help you if you want a cube Allah to help. Secondly, you have to declare them when the config yeah well there. If.

J

A

Can make these schedules containers.

M

Cuba wouldn't know about them.

A

Of course like, if you don't use the infrastructure, yes, they won't live right, yeah that.

E

Is my my despite the biggest downside, even more than resource attribution, yeah.

F

E

Lifecycle management and that's why I'm generally, not in favor of the the binary upgrade out of the end of the platform, but.

A

Why like, why, wouldn't you be glad and the invoice is containers in the demon said? Why would you no no.

M

No, let's say we're still living in pod right. Well now we have this in a container and it goes hey. Node inject my envoy. So start my own boy run an injected into me right. So we can do that right. You can exact it into the sea group, but kubernetes won't know about it. Cause it's not declared I'll.

A

See you don't one so you're still wanna run and boy with the pod in way.

M

F

Have a container container.

A

Okay, I thought you were talking about when envoy is like running in the demons vampires, no, so we're in this case. We.

E

Always a lot of debug ability, I think in that world, like I, can't.

A

Use the standard commands yeah I, don't.

E

K

A

Advantage I mean, except for the binary okay.

M

So we are suddenly saying we are constraining ourselves to just the DNA container. All it does. Is it asked the demon to do the privilege, nothing else for.

K

Now yeah I think, but we.

F

Couldn't use that I mean I would say that, instead of just simply saying set up the IP tables, if we find out the way by which we can have a worldly amount, where we can stick the Envoy that we want and then, like you know, defy the code and piloted in distance I started actually goes to slash slash my own slash, unwind boots that then it actually provides us a nice way by which all we have to do is Mount a new version of envoy in the same volume out I mean I'd.

F

Let my and pilot again simply needs to get and and the equal of an eye out pill or some kind of a ping that basically our allows it to do or nines nice hot restart. Oh, it's Auto I saw a new version of an why. Let me just go and.

M

Start within the container, that's fine.

F

Right but least I within that container, and so what I'm saying is that for that we can actually use this mechanism by which we we share a mount I, don't know whatever the Cooper.

B

F

But such that on.

B

F

Is actually in that mount and that actually allows people to do a gradual toward on a per node basis, because they can actually decide that this node is gonna. Get this mount and then the new version of one way and all the pilot egg agents on that node will actually start picking up the new version of on when doing a hot restart of the the respective invoice, which gives that nice, gradual, rollout thing that we actually wanted.

F

A

Done something similar with does like in mini tube I was mounting a volume in my mini cube and then I had got adult like in running, and the process like the pilot compiled locally and I were just replacing the binary. Basically, and you know feeling.

K

A

Getting the new binary so that that should work right for it for upgrades. If we go this way right.

M

Sorry off razor that reinjection is, is a useful thing that we want to get to. Alright, we probably flogged this horse of that look. Clearly, we need a doc outlining something options and.

A

An owner owner.

M

F

Thought I nominate, lose Ryan.

E

Physically far away.

E

Busy turns out.

M

Alright, well anyway, we need an owner, her at least for viewing Bob Bailey has so, and that probably shouldn't be me.

E

Yeah I can I can help out with the reviews there. Okay.

A

Here with us, no.

E

He had the leave earlier unfortunate.

A

Ok, yes, you'll follow. Thank you. Problem sent an email, yeah.

J

So I don't have time to sandra.

E

J

Yeah, this is Andrew box from her head as well, so I'll I'll hang up with Rafaela after the meeting. Okay.

A

Great yeah yeah.

E

I don't have time to show around the codebase, unfortunately, but I am happy to review peers.

J

E

They come in and then help out with that, but yeah, okay, great.

M

M

A

Yeah I will definitely need to follow up on that right, so this is or I was, thank you everybody and enjoy your lunch. If you have a lunch break now and if not enjoy the rest of the day, bye see you next time I do it. Thank.

J