Apache Mesos Containerization Working Group, 16 Nov 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GMT 2017-11-16 Containerization WG

Description

Agenda and Notes:
https://docs.google.com/a/mesosphere.io/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit?usp=drive_web

A

A

A

A

A

A

A

A

A

Yeah, that's away for a few minutes. I think these guys trying to join.

A

Yeah I think the truth is running a few minutes. Late I think: let's wait for one more minutes and then like I started, and these guys will join you later.

B

All right that weather yeah.

A

All right, James, James I think we should get started. Maybe we should start from your stuff. First I'd like to kind of go through your implementation, whatever design for the user, hey, Gilbert, hey I, think geez. How is gonna be a little late, so, let's start with James first and then see. Jetton can join he's gonna be a few minutes, but okay yeah, sir Sir Emily and Norris okay, so James do I like sugar screen. Okay I saw that sure.

C

A

C

So I mean I, have you know, code um basic stuff, working? It's pretty straightforward. It's just like in there to actually fit user name space, it's working because most of the infrastructure is fine.

C

So as far as implementation goes, the goals are. Keep it simple. Let's not do anything too, fancy there's one and we're going to focus on user name spaces for with regard to security, domains, we're naughty but excluding any sort of ID mapping for now two reasons for that. Firstly, I don't have a use case for it and, secondly, it makes it a lot harder. I don't want to do with a yes, so in in the model, we're kind of distinguishing three level. Three then no necessary nested, but basically three API levels of of these namespaces.

C

If the root namespace, which is then, then the default user name space with the hardest, we have the task group. The task group namespace, which is a user name space that resource will create containers within. So, although it's going to be the parent namespace of you know their containers, map, namespace and network, no space.

A

And that kind of stuff, so James you way mean task, group name say space. You essentially means our containers than name sorry names yeah, because there might be multiple tasks or for a single, a secure, I.

C

Guess no no I see a single executor would be the work of a container tree and a container trees. A task group, oh.

A

Right I think, oh when I say task group, okay, yeah I, think the terminology is kind of a little different because task group inside may. So it's really just a bunch of tasks and they can send multiple task groups to an executor.

C

It's supported the terminology. Is you like the metals terminology you submit down very clearly so I was explaining this to to a colleague using the term container tree, and we um we found. That was a confusing terminology in that conversation. So so today, I'm trying today I'm trying tasks. Okay, so we.

B

C

Maybe container tree but.

B

Whatever distinction between tasks, groups and the executor, what really one is hard right, which has a clear definition and in this tree, however, it is not to me so I think you right pod right, yeah,.

C

We could say pod yeah it'd, be a tough group in this knowledge. Okay,.

A

Yeah I mean I think we're trying to kind of decoupled the container hierarchy from the executor task hierarchy. One of the couple of these two but like some of them are like tied together, for example, container it's always tied to in the executor yeah anymore,.

C

A

Anymore, though, like was the standalone container being to do so. No I'm sorry I mean if so, maybe this maybe this indicates.

C

That there's room for some sort of better description of the me sauce model, great including terminology, so that we can discuss these things. You know okay anyway, so you go so there's the container, basically there's the container level namespace and then there's what I'm calling a private men's space, which is, which is a leaf and the private name space doesn't have any other name spaces created within it. Certainly that's a leaf in terms of the security model.

C

So, basically, you said this is an isolator when you enable the isolator it automatic, it just constructs the task main spaces, the transcript user name, space, the maintainer tree namespace, and you there's two two flags in Linux info: one is private user name space, in which case there's two levels: you you get the container tree M space and when your task is launched, the targets launched into its own private username space, and that is a leaf namespace that doesn't contain any other kernel objects.

C

Okay, so why didn't get two flags.

C

What sorry the share use, namespace, yeah I, think I'm trying to think of why I'm you so the shapes are the share, use namespace, server, open question at the bottom for that um the show username space, okay, the use case for that one is we want to have. We want to be able to have tasks that join the root namespace and don't get a separate container, make space.

C

Because we run is still spirit, support tasks that actually do manipulate those. Oh.

A

C

A

But if I mean, if that bullying is set to fall, so you would that be the same first one. So I can't confuse like, if you said sure, usernames be too fast. Will that be the same as a private using his face.

C

A

C

Because because the distinction is, that is what private username space means is that my sauce isn't going. Is that the username space, your final task London, will not be, will not be the parent using them space of any other kernel objects.

C

So, in terms of their mentor internal kernel, men's space tree aha,.

A

C

No than that this private user names first won't be the parent of anything, so it won't. So when you construct a container, you enter that you enter their username space and the mountain home space, and maybe the milkmen and the previous place, and all this other stuff and the name spec their user name. Space is the parent of all these other namespaces right mm-hm.

C

So that means that tasks in the same user name space can have privilege over the other namespaces once they will have because they share a common parent. Oh.

A

C

A

Two in there so.

C

When you have a private user name, space you're running a task within a user name space. That is not the parent of any other namespace, which means that it's not possible for that task. To have privilege over any of the other namespaces that that we use to construct the container.

A

Okay, what's the default behavior, if it don't set any of those flags, the.

C

Default behavior is to run the task in the container user name, space.

C

A

What yeah, I guess like God little confused about that container and TAS eros yeah? You keep mentioning tasks. Okay, so.

C

The tops of the tasks, the terminology, somebody else, Oh change,.

A

C

A container yeah.

A

C

No space in the task- oh okay,.

A

Are you talking about the which ethic here? Are you talking about or you're just talking about so.

C

What I mean by what I mean by user tasks? Is the final program is launched into the container, so it's either a binary from the from beside the binary that is running or it's an executor. A binary are in a secure, that's the command.

C

So if it's, if it's an executor container, it's the executor, if it's a mr. container, it's the container command.

D

So why do we need to distinguish the contingent in user name space and the tarsiers and in space could it could it be identical um so.

C

No, so we need to distinguish them because what matters in using namespaces, so these namespaces are defining a security domain, and then we have a private when we when we run the task in a private using namespace here. What we want is we want. We want to arrange, so we will arrange for the tasks to run in a user name space.

C

That is not the parent of any other user names roses, because in order to have privilege over some namespaced kernel resource your task and the resource that were checking that you're having privilege against needs to share a common user name space. So what private using namespace means is that we guarantee that you that there is no other kernel resource that has your user name space in common.

C

So if, for example, you are able to say you're running a regular program, it's my private user name space and you were able to acquire some capability right, you're able to acquire some privilege, which was unexpected, um that wouldn't let you manipulate the user name spaces that we use to construct the container because those user name space. Those those namespaces like the mountain named space, for example, is owned by a different user name space. So you don't share a common path. Oh.

A

Okay, so maybe like it will be easier if you explained that the whole flow you're saying that for the container, oh okay, you create user name, if the container and and then and then, if you set this private user name a flag to chew, we actually like launched a seizure or the task. You actually put that into a another separate user name space. Maybe.

C

Maybe you've heard about this some of this explanation, so the container name space protect us to host the task name. Space protects the container right so.

A

That's kind of clear okay.

A

Then the question is what oh I've seen the contrary: name. Space protects the host and the task names to protect.

A

Yeah I saw, but but why not just treat container namespace the same as root like I, don't think we're in any code there, alright! So like we don't have any process of launching that. Do we, because in a secures launch incident user task name spare, it's not the audacity launching Larry.

C

A

C

The first implementation I have everything will choose inside the container namespace uh-huh, that's the default behavior. The default I have you for the outside. It used to launch everything into the container namespace uh-huh, and the issue is that is I. Guess there's! No. We sure that it's it's fine and it reads it's. It's gonna protect your host. It's.

A

Gonna protect the host, but in it's likely that some tasks container might affect some other tasks container because they're in the same user name, space, yeah. What you saying yes,.

C

And that might be what you want. Oh I say.

A

C

Have it so in the current implementation and I'm, probably gonna argue that so yes, the current implementation has a single user name space for the entire container tree.

C

Mmm-Hmm I am NOT, so this is not open questions and I'm, not convinced that there's like a I, don't really have a computer in this case to have different container containing namespaces yeah.

A

C

The tree I see.

A

Yeah, it's kind of you're saying if you I mean if you want eyes essentially you're saying that, like some tasks in that part is not trusted and some some other task is, can have sensitive information. Maybe we do have that use case, but yeah I don't see that for now, yeah.

C

So you know you can say that you can say: hey I when I have I have my tasks. It's in the container. The container has a user name space and the task has privilege within that container and there's that's fine, so, okay, the tasks can can do things to the container okay yeah um and then you could say well, I have this task and it's it can tell you, but I really don't want it. I really, don't want it to do anything to the container. Well, they come back to use Case Squad at that space.

C

A

So is that mean like we don't even need these two flags.

C

We could I mean we could assert that everything gets a pilot using in space that feels fairly restrictive, though oh I.

A

See you still continue to join the hosts user name space. Is that so there's two so in basically.

C

There's two things that joke spectacle: they're actually one so there's a private user name space which is I'm running something that I really don't have any compliments in right. I, don't want this to have any I want to protect both the container and the host from this task.

C

That's what private isn't any space means and then goes then root. Username space is the opposite. That's what I'm running a task where I do want it to have privilege on the host itself. That's yeah! Okay,.

E

C

That and that's in there strawman here I've said that's. You would use the cheese namespace flag yeah so whether that whether we call wet share using them space or we using them space open to suggestions, share using them spaces more consistent, API Williams.

C

A

Does that mean for net space? Have you thought about this like? What do you mean? What does that mean if we say how many launch and nasty container set sure using a 32? Is that sure was the parent container well yeah.

C

We would have to make that if we call it share using them status, we would have to make the semantic to that consistent with the share. A big names go swag, okay, I.

A

See, okay, I can't understand a flag. I can't happen. It understand assure user name space, the private user name space, okay, so it's a. Let me put thank you sweat, so so what if you set share user names? Youtube a private unity space to true as well? Is that a failure, validation, failure, no.

C

That would be fun, so I mean, except you said, share usernames place to true and you're the root of the container when your container will be constructed in the we're continually constructed in the root username space and your task will be launched into its own pilot using namespace. Oh I,.

A

See it's okay,.

B

Sorry, I missed a part. Can I ask another question, so the Linux info is attached to the container info, which is used whenever you launch a container with whether that is an executor container or the you nest container. So so that's why in there we even with a share pinning space thing, you only say the parent versus, not you, you don't say the root, because you don't know your your accessible. If the root is accessible to you right, if it's, if you're already under the nested container, then you preparing is already not root. Can.

A

Is where it's still available to you, my understanding, sir Sir James, correct me if I was wrong. My understanding of this is usernames is like a special namespace. So there is a relationship between like other namespaces compared to like say you can say a mal namespace is it's under that particular user name space because of dad that kind of complicated the flag on that?

A

We cannot do similar as to to what we do for like 10 min space right, because you do have a situation where, like you want you want on like so you have two nested container under a top-level container, you own these two nested container to not affect each other. It's wait hold on.

C

So so I think to answer your question.

C

Yes, it works the way. Yes, the supreme work. Fine. So if yours say, if we implement share username space, then what that this manage that would be join the username space of your logical part. So, if you're an executive container, you will join the roof username space. If you're a mr. container, you would join the username space of your executor.

B

Right, yes, that makes sense to me and.

C

If it's false, then you will get up in your container.

B

C

Will get a new container.

C

For your container.

A

Okay, I got two countries, so you're, saying.

F

A

G

So I guess the question is.

C

A

Pitting space we only have one flat which is really share with your parent spending space. Can we do the same for username space or not know.

C

The report, because we have this extra layer, we have the earth and I- have an extra layer of many spaces to protect the container from the task and the situation doesn't arise in namespaces.

B

Right so the distinction between a container and the task is that what you said, the colonel objects versus only the task process, and so, when you clone, let's say for launching a new container and you clone- and you have this whole list of names is namespace, is specified and.

E

B

Username saves in it and then I guess: that's your container username and container username space right, which is which has other names, but this is when you clone and then, when you clone you just yeah that makes us container rights are launched there, which then would fork exact your image processed image. So what else is this? How do you achieve this private username space after that? Basically, starting from that point on, how do you do additional isolation so.

C

This in when my sauce is launching a task there's the climb which happens in the launcher, but then later on, you run the containerize a lot like being equals containerize Aloha, and this is the final stage to actually executing the command.

C

At that point, when you execute the command, you have to construct an e in the using namespace. My.

A

Point is why why do we need to do that? As you mean that, like let's say you re creating user names for the container, why do you need another user name space for the task? Given that the container is our basis, I mean it's not executing anything, because because the container, the computer.

C

So that the container is is made out of namespaces right mm-hmm, the parent of those namespaces is a user name space right mm-hmm.

C

So when I launched a task, when I run a program whose pod is the same user name space as the namespaces that are constructing their container, then that task can acquire privilege over the container over a.

A

C

A

Because they share a common because there because they share common using namespace. And why is that in the issue? Because they can.

C

Because if I don't trust the task, if I, don't trust the task say I, say I'm, the suspicious person and I think that this task might be able to acquire privilege that I didn't explicitly grant it.

C

If I'm suspicious, then what I would do is I would say well, but I want to do you say: I, say hey. The task has its own user name space and that user name space is not the parent of any other name spaces in the container. So if the task was able to acquire privilege, it wouldn't have any privilege over the container because they have disjoint username spaces.

A

Yeah bye, sorry what I thought like when you want okay, I'm thinking about the like, like we want to launch under the task in that and I say the top-level continuous I see here is container and.

C

A

The a secure wants to launch a task container under that executor the room, that's the container, API or or directly just for kids AK.

A

If it's well, if it's default, I secure just hit the agent API to create a stick container, so the NASA continue is under a different user name. Space. Alright yeah, then Daniel, lock that the container will actually like focus at the actual task in that the same real name space as that nasty container yeah.

A

So that task in that NASA container won't be able to do any change, because it's still containing a continued innate space unless oh, okay, I guess the crazy man saw, is unless there's another nasty container, underneath that the level two nasty container like this is.

C

This this is the same case. All right, you have a you, have a you as a user name space, which is the parent of the container right mm-hmm. If the Todd. If the task has that, has a common parent as the container, then the task and acquire privilege over the container, it doesn't matter what how deeply the container is. Mister mister borges calls make sure I see.

A

No I sort of get it so you're saying like it, doesn't make sense for the task itself to get the privilege over the container, because there might be some others has launched under that same container, and you don't want to want that to affect the other. Only it's even if we just go Sanga.

C

So we say we assert that every container gets its own user name, space mm-hmm.

A

C

An unprivileged task, I, don't want I want to use, have a I want the head for an unfinished task for one. I have I want that task, to have a unique user name space so that it can't acquire any privilege over the container, because I want to reduce the risk of it being oughta break out. The.

A

Consequence of a break is not that big I don't know like what what that task can do, because still containing that containers using a space.

C

But I don't want to I, don't want to have that can do I. Don't want to have that talk. I, see, manipulate the container I see.

A

Okay, so you just might be super okay. Okay, now I got it okay, so.

C

The distinction is, if it's in a private user name space, you have less ability to. You have the most ability to alter your container so.

B

Is these user process in that container, but not in that task? Basically, just the missus containerized lunch helper yeah, which has a yes.

E

Just can you hear me? Yes, okay, I was doing the comparison with the the pit namespace actually in this isolate others. Only two level right like what you try to introduce here, is a three level name facing feature like your route versus container versus user and for the p.m. space, for example, for example, there is only two level right, no paintings.

A

Mystic cream it there can be multiple levels, so we have a flag that I share with parents. So if you set that teacher the pinyons we will share with the parent continue, we.

F

Can have more than two lab.

A

Of Nursing containers, okay,.

E

But I mean you can only do the same feature that here with the the share share parent or something to.

C

E

You cannot I mean create and also a pin names before the task, probably separated from the executor on the container and so on. What.

A

Depends on which a secures if you use default a secured and that's separate, but if you use the old style command s okay, so the executor is kind of like legacy and secured do its own for kids. A can only really really is leverage the username space mm-hm. Sorry, not the pen. Sorry, don't create a printing straight for the task.

A

E

So the behavior is different if I change, yeah I.

A

Think that the goal is trying to move to the default, a severe scenario and whenever you want to create container nasty container, you had need to hit the agent API to get that created. And then agent well properly set up the nesting for the tenant space.

E

Okay, because in a in my vision, I mean there, there is a surely of similarities between user name places and UID name species, for example: ok, ID namespaces, are you Oh, like you, can as a naming convention generalize the fact that there is child and parents, because we were talking about nested, containerization and etc? So why not changing the naming instead of fruit, calling parent and self? You know that's coming uh tried. Oh yeah I think anything.

A

I think if I miss Roncalli, that's the intentions that the footage.

E

C

A

And a space to share with parents, especially yeah.

C

Yeah, yes, you're, correct, there's this third layer right so change, namespaces, parent and child which we have the pig, is namespace additional one, but we want for use namespaces.

A

So James that brings the interesting crashing. So we do the same for Ken a space.

C

No, you can't do this only pin namespace, but because pit namespace basically requires an amount of rock and everything so for a security perspective.

A

I first, you want this feature because I can't have the same argument saying that right, like just ignore the implementation details like from security perspective, I pretty well my test, you not be able to see containers pit containing like the pin a space of the container like the in the process. I guess we probably need to allow these.

C

Not see the unit process, you have to have the new process well,.

A

I wanna hold on so I want any process to be my tasks right, for example, a knowledge system thing yeah.

E

The chosen I'm going to a basically okay.

A

I feel this is the same thing. Maybe we should should should think more high.

D

A

This may be foot, do the same for penance. Oh I love those basically I. Think for pennies based people. Some people want say: hey I, don't want to manage my own index. Please do anything for me or some people just prefer I'd say I want to do my own in their time on our system V, so we purchase support. Oh I think we definitely to support both yeah I. Think.

C

I agree with the use case of being in it or a happy able to do your own in it, but I I, don't know that this is the right model for okay. If that's I think you can probably do that within the existing model of parent and child feed, namespaces yeah.

A

If you think that's not the right model, I found I'm gonna question like why you think this is the right model for use in a space.

C

Because user name spaces are different, so we can talk about some I, don't know what the API looks like we can I mean we can talk about what okay I doesn't have to be a flag. The API could be something different, but ultimately you want to. You can separate these things out in a way that's completely transparent. So.

B

I guess maybe what she was asking has to do with what I was coming in the dog, which is, and can you achieve the same thing with the dish, snow levels of nesting or nesting, a sense of mesas containerize, ur lunch or in this finer grained different actions to do when you're launching this level of container I.

F

Not sure I know what you mean because you so you.

B

See that and so what so I guess it's probably helpful to when we're talking about writing about container launches to two separate that have fun may so the executor and task, or whatever the basis terminology and workflow, is and basically just think about.

B

You have this container to launch and you want to put a process into the container forget about tasks, because, ultimately, this thing can be arbitrarily nested right and even, if there's notes since to use it that way, it's I guess one or a little bit confused when we're talking about containers and the tasks at the same time, whether so, for example, if you, the health checker model, which is you'll, launch an s2 container by the default container by the default executor.

B

But then your house checker, is nested under not the executor level, but under the task level. That's a third level. So all of these nesting levels. How do they affect your reasoning of security and the use of private versus shared I? Think, okay,.

C

So it may sauce when you're talking about Nesta containers, it's just logical. Mr. right, you just have I mean yes, it's just a logical nesting of containers. Oh this is some internal notion of opportunist and when you have a private, the private user names face here, it's not a container. What you're doing is your protect your leveraging, these namespace security model to protect the container from a task. So it's a completely different thing.

B

But I guess: could the question of you guides? Okay, so you for concern about this level, you not protected from the level down. Could you like you launch a container nest container which for security you launch another nested level and in between levels you have shared being false?

B

Would out achieve anything, no.

C

It doesn't do anything because the container they on the container is constructed out of kernel objects. The parent of the kernel objects is a user name space. You launch a task with the same user name: space parent. It doesn't matter like what you've done internally to my sauce, to keep track of this logical tree of containers. What matters is the relationship between the kernel objects.

C

So you can say you can make as many nested hierarchy making many nested containers in the me sauce nested container model as you want, but it doesn't have any effect on like how the actual security properties of the system.

A

Okay, so I think, like I, think we should per set up a follow-up meeting on this I think, oh, maybe like we should always I feel like I important to do some reading on the stock first before I call me on this anyway. So um so James do you have anything more than this API No.

C

So down the bottom, there's links to the jurors I have a basic I, have an implementation. Four eight one, four two and I'm working some basic test cases, a.

A

Shared the dog to the the slack Channel that container migration China sure I haven't seen this stuff before sure. I only wrote it this morning, okay sounds good, so uh I've heard like the next step, is true to comment on this dog and think think more on this I don't think like the one hour. It's something like half an hour is sufficient for this, like for any to think more on. I thought it would be ten minutes. Okay, yeah, sorry about that.

A

I think we do have another agenda, so I want to switch a different agenda and we can't follow up on this, maybe next time or like during the next week or next week. I do want to comment on this cool, so I'm sure sorry on the line. So maybe we should switch to the the GC stuff.

G

G

So we're gonna just do a demo, quick demo and vineos.

D

I guess we can look at them and just go couple minutes and then I we can check out the the auto decide in the you know, a decider, the auto GC design.

G

G

We have right now for the HTTP work, so right now, I mean most most of our things inside the container Iser from container i0 to the unique provisioner to the dollar store. So when people right now, we are able to trigger a image GC for the docker images through experimental API agent.

G

Execute run like two tasks on the machine on the end, which just sticks forever when I'm using the default. You punko, docker image and other, would be using my secret of your image.

G

Block see, we are fetching all the contain other related images. Let them finish okay, and we can't use this. This HTTP request it pretty much just called call operate. Api called print images on the agent.

G

Okay, it said through I think in this case, in this case Gospels tasks, both images at the consider still active, so we are not going to delete any layer right now in the print out now. Let me stop wonder the task. The one stopped would be Ubuntu.

G

They try again.

G

Go back to the agent rock. We see that I think we see that from the a from the Braavos login, you can see that we decided to leave the bunker players from the system and we also I think there's also log lines here say we we are marking a cup of tea, I, think there's also there's also actually else. We logs indicating that certain logs so other layers that the Mexico image is using a marked as kept during the GC. Oh I think it is.

G

Bring another DC I.

G

Think we should.

G

G

This should be empty. Okay,.

C

G

This previously, they should have a player's here, cool.

A

mmm So just one question: so the way do the HTTP API that the response is always 200: okay, yeah.

G

There is only one one chance it can get of. 400 is why we do not have other. So there was a couple of dependency patches requiring me to check, find the relative information yeah.

E

G

Information incomplete, say: okay, we cannot safely do an image TC right now in the store, I. Think.

A

Okay and the like, do we have a way to show at how many layers being deleted? We don't have that right now it can be trivially. I did as a conquer. Any metrics for dad's had the variance okay, Zito.

C

You is there any GC without it me calling the API that.

G

Is what Gilbert is proposing? We discussed today like to automatically come up with a policy to do a automatic TC based on certain policy or heuristic in the agent yeah.

C

I mean I'm thinking for my use case, like how I wouldn't know when to call it, so the best I would do was maybe set up a cron job to call it all. The time which seems like my socks could just do that. For me, yeah.

A

So that's what we want to discuss next I think Gilbert. Do you want this yeah thanks just how I think that's great.

B

Yeah at least right now you know that if the image is which no longer is well, oh yeah.

A

Well, I think so so so James to your question. So one simple way: it's like you can minor the the this compressor if it reach like a certain threshold, you can just call that API how.

C

Would I monitor the dis pressure you just do.

D

C

Now that there's been meat sauce has a notion of dis pressure, so.

A

That's what we.

C

A

C

Don't I mean I can see if I could monitor and if Mikko tells me about it, but right otherwise, I'm just reinventing yeah.

A

Sure so yeah, that's that's! Gilbert wants to discuss.

D

It's me yeah yeah, so there's only step like 10 minutes, so maybe we can quickly go through the display of the OGC. It is a basic one Oh before before that. One thing to note, it is like, if you want to use the GC for mazes continuously, you have to and your cluster it up way from a previous version to 1.5 or later you have to drink all the container on the agent. Otherwise, the the GC gonna be disabled.

D

This yeah, where we have no way to like support ever competitive capability to for the legacy running containers well,.

A

It's well you're, saying, like you, ain't that that endpoint cannot be used because the endpoint will return 400. Is that what you're saying yeah.

D

A

G

Both in find the whatever auto TC, we would add, there will not be. They cannot be safely performed on the on the container either because yeah.

A

And the reason you did the reason for that is because, because you don't have enough checkpoint information in the previous versions, is that reason: correct, oh okay, yeah, so weak on the document that yeah definitely document this.

D

A

Any question no I do want to go. Go through the auto GC policy thing yeah. Let.

D

Me share my screen: okay,.

D

Okay, let's see my screen yep.

C

D

So this could be a hat. We already have a discussion about the auto GC, so it's it's hard to. It can take some time to decide truly auto GC. So right now here the auto here means it is basic support for auto GC, and we just introduced on here st Roger in the agent in the agent code, to allow operator to specify some agent flag to so that we can trigger the image garbage crashing theoretically and the the discussion start from the what's the policy we want to trigger their GC.

D

It can be tied based or size based, and it can be also be a combination of then, if some other factors, so maybe in the future, when we introduce the Remy increment, decide the unified artifact store, we can introduce some like Auto juicy policy photograph to allow user to divide GC policy for image layers and for now we we just critical to this to to to option and sighs bass makes more sense to us because for Thai based, if someone just want to like trigger the GPR periodically, why not just deploy the Kong job and then chicken?

D

Did you see weekly or monthly, so the auto one we prepare to go with the size based and so right now? If, if you look at you guys, look at the currents and execute the same box GC in our you know, ask you to sandbox GC. We will schedule sandbox for garbage collection, and then we have to agent that can show that one it is the same. Dosage is a heirloom. Another is the watch interval and we can introduce similar mechanism for the image layers.

D

And basically, we have two options here in a from our decide when it's like introduced to a jump rack to control the Headroom and interval of the GC, the Headroom means the harem means, like the distress show, because we, the an agent, monitor the this usage from the dollar store directory and once a rich, specific percentage.

D

So, for example, the head Jong, it can be like point- can be zero to one and if it's like one, it means like if the usage of the disk up more than get up to like 90 percent, it will trigger the GC automatically, and the watch interval means like how frequent me check with this usage, and so this to it has. It has to be like both set or not set, and we also have another cracker control the exclusive excluded image. So it means like it just matched the operator API.

D

So right now the operator API. We did not implement that yet, but in the future we will add an excluded image. It may it's an image. Even it is not used by any continued on the agent, but we will still keep it and catch it on the agent. So yeah, no.

B

No I guess my question is the head room and all of the calculation is that it based on a size or that basically, the remaining free, this free disk space of that volume, but that disk I thought there is gonna, be your like a size for the disk image for they for the image cache. It is not size. It's the percentage of the this usage of the entire disk usage right yeah.

B

But what, if I, don't want the entire disk to be used for the images so I have to set up Headroom based on yeah.

A

So I think, then you have to calculate the heroine in a different way. Based on what right now we don't have a mechanism to enforce the disk usage for a store right. So right now we don't have a way to yeah I.

B

See I see I said so I. Guess it's not like this. This wouldn't work, but I can see that this is. This is kind of modeled after the sandbox. Do you see, and now the fetcher cache, even though I'm sure we want to improve, but at least the fetcher cache you have a have a designated size and you, you know the size of the things you put in there and and based on least recent use.

B

Of course, there's more things to consider here, but you basically see how far away are you from that size and you do GCC based on that and not that that the disk rusher like like sandbox GC, because for that you want to leave them as long as possible because for future investigation you might need these artifacts, and this cache looks to be more like the fetcher cache which, if it's no longer used.

B

Then and tension a desire of retaining it for a long period of time, I guess it's not there. So there are two ways of thinking about this right. Yeah.

A

I think that's a good point. Yeah I think like yeah yeah, that's a good point to take a look at the fattier cache flags. Is there a size for the fashion cache and then, but we don't have a GC? Do we have a GC for a geocache? No, yes, it just.

D

A

Just needs me recently used.

D

B

Yeah I think for fresher cache. It's easy because there's no there's no.

B

The package is the package: there's not like layers, which could be shared a lot, so that sure has a venture cache as the size and the easy thing is things you're. Putting the printer cache it's easy to calculate its size. So you basically just store the size of these of stuff you put in the patrick hussion without watching the disk, because because you know or once when you are fetching it down and then you just do to do the math internally.

A

Writes the GC policy is GC. Mccannon is slightly different, but yeah I agree that we should definitely look into a house yeah, okay, yeah I think we should definitely look into like the Fancher cache size way, because we haven't discussed that in fact. Yes,.

C

Does in general, does the image please say bring us closer to being able to unify images and artifact and fetch artifacts into a.

A

No, no really I think that the goal is trying to wrap this up ASAP and then move the the developer resource to to have a artifact artifact store for those fabrication layers, and it would put more energy there. So this is like more like a stopgap solution. Yeah to run this in production, yeah.

D

And- and we do have the plan when let me introduce the unified works door, we my move, the federal cash and permissions are together and to have the unified restore to control tinna of continued image and virtual cash uri. So and then we do a hash for ten together and at that time of you introduce another GC way. Another GC with different agent flag to can show. So it might be a good hint to start to start to think about, like yeah.

A

D

Can show the size of the.

A

Chemically difficulty is like for to do the factory way. Is we don't really know the size of the layer?

A

It's not like artifacts that you store those artifacts. You know the size of the file, but for this one, the layer and once you enter it, you don't really know how you can.

B

A

That doesn't change right, so yeah.

B

A

I, don't wanna, do it D you I mean how do you get the size for that entire layers like it's, it's a trade-off right so like either. You have some way to imaginary figure out based on tar file and the compression ratio to calculate the the size of the layer, or you do tu, which is kind of expensive. If the layer is huge and I expect, it's gonna take a long.

G

Because we still need to expand the content of the directory anyway, I'm learning imagine to you. This is probably if it is a linear to the size of the layer. It's just slightly higher.

G

A

So my point, it's like we need to do more investigation, see how easy it is to do that. It's not as easy as just getting a side of the file.

C

There's gonna be proportional to the number of files not to the sides right.

A

D

Files sure, yes and Tartu, I guess and for OCI in the future in innovative story. If the OCI support me, we have the size of each layers, I.

A

Think that's compressed, so that's hard like not the size of entire layers, so we need to still think about how to solve that issue.

G

Some previous idea is trying to slowly bridge the two things related.

G

You know the start, other things of endo storm, yeah right.

B

Yeah so I guess yeah. We should. We should think about that. But even though I guess the point here is not about, you know whether that can be unified or not it's just about so right now, we're basically relying on the operator or the user to to be able to, or for multiple discs, to do heart level, heart, hardware, level, isolation of discs- and you set aside this disk for this purpose and that for that purpose, but mm-hmm other.

A

Things yes, which.

B

Uses this space, what if I have a lot yeah? Okay,.

A

I guess you get my point: yeah yeah, so I get the point, so I think there's a lot more I need to do, for example, the upper layer for overlay. We need to cap the size. That's another thing that we want to do things like this, so it's this is just like a stopgap solution to make it production kind of production ready. If you don't have GC like even very simple GC policy, then it's not production, already, I! Think yeah.

C

Have you considered a API to tell you if GC is gonna, do anything like some kind of API to say? Is there anything even available for GC? Is that something that would be useful in your workflow? Oh yeah,.

A

So that's that's a good point, so I mean don't I. Think we don't have that API right now right go! Do you don't, but that is pretty straightforward to expose I guess yeah.

D

I think what what you're! Looking for! What Jones.

B

D

It's the I thought.

B

You once you previously mentioned the metrics at least the metrics in a faulty I. Would it be more useful, yeah yeah.

D

It is we can, we can definitely introduce the API, but if you involves an interface, some new interface in the continuous area which I might um light, because because we don't have the image management management at this point and then if we want to introduce this and like show GC ago layer or show all layers, it is like kind of like we can introduce some temporary solution. So why not? We just do all together. Do the image management all together in the unified artifact store in in the near future.

D

Let me support the OCI, so that can be something we can do in the future. Yeah.

A

So October I think like having some metrics. If it's not hard to die, those mattress I would prefer just adding those metrics. It's always good to have more metrics than having like last matches yeah if it's easy, but if it's hard you think it's hard, then it's okay I think we should spend more energy on the unifier artifact store, because that's the direction that we think it's makes sense in the future. Yeah.

D

Yes, so for metrics it sounds good to me, but for a funny helper new operator, API to list all layers, I will say like we can definitely do it with the unifier data store, yeah, I, think matrix.

A

If it's easier to get like a cage that you can easily get just like all the layers, I can potentially delete a number of layers. You can tell these things like the style, be useful. Exactly.

B

C

I think basic metrics that would serve that sort of purpose would be like how how much how much he owed me how much you're consuming you know as a gauge how many times you ran their GC, how many things cleaned up on each TC run that kind of thing right, we're.

C

A

Yep all right sounds good thanks, guys, I think we were over time thanks everyone for attending so on. So we part have a demo for a standalone Pantera next next time and because the all the killer are committed and you wouldn't call up down on the user, make space discussion and this one too, okay, I guess you haven't seen Alexander's, you have to send out design talks with the community, and then we can iterate on that which decide. Oh the auto GC the flag, you wanna, add yeah yeah sure. Maybe we will send the elevation.

A

All right sounds good thanks guys thanks everyone see you guys next time, Thanks see ya. Alright, bye.