Apache Mesos Containerization Working Group, 30 Nov 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GMT 2017-11-30 Containerization WG

Description

Agenda and Notes: https://docs.google.com/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit?usp=sharing

A

B

You guys, how are you hi.

B

That's wait for a few minutes, so just how have you landed the GC stuff.

C

There is one one operator, API change that has not been landed yet, but I think we change this all very short. There is a one-liner bug, fix that's being sent out last night, okay,.

A

C

Other than that cool I think yeah I think the the auto GCE in AJ and I guess Gilbert should be working on that yeah.

B

I think that's what he's working on right now: okay,.

B

B

B

B

Okay, like I, share your screen like a I'll open. It talk, sorry you're, muted, yes,.

C

Let me do that. Yeah.

C

This visible yeah.

B

Let's wait for one more minutes and then get started and some folks may be late. There's no way for them.

B

B

B

B

B

All right so just talking, let's get started so. Can you tell me a little bit like what you plan to do? I.

C

Am trying to use this time to decide what to do I think I lay out the two designs here: I, don't have strong opinion either of them. I guess want to see. What's.

B

The motivation like, what's the highlight more, like a goal you want to achieve the goal.

C

Is to making sure at least four I mean, let me be clear: executors control, executors containers, uh-huh.

B

C

Any executor container I want to making sure that regardless body a container is doing, they cannot write infinite amount of data on to the disk. Okay,.

B

Are you guys using a stick like custom executors? We.

C

Are not yet we plan to write one okay, but I, don't.

C

C

Hello, yeah, hey I'm here, because, regardless of the default executors or custom executors, if you long for a token image right now, yeah as long as you find a analyze, the rest for path, that's not bound head would end down the route for system yeah.

B

So I guess I'm curious. Why, like a you're limited to just a secure containers, not all.

C

Okay, let me let me drop it. I think I secure the container is really urgent. We don't have a nice kid containers. Yes, you.

B

Guys launched a screen. Are you using command executor right now? We.

C

Are is, it makes a mitra, we use both their most command executor, edit, before the executor wheels all over bit different setup. Okay,.

B

Done it, okay, I see actually.

C

Let me just make it a go there. This should work for both executors container nested container and standalone container yeah. All.

B

The containers, okay, so that's so that's clear, okay, mmm okay! So the other thing it's like um what doctor does right now: yeah doesn't.

C

Do anything okay, which is the so this is the this- is the kind of the really I think this is the last piece of seeing that requires operation in operator intervention in our cluster. An isolation stack CPU memory. Noise is annoying, but they they kind of. They need to be there. They kind of had know differently, but this space leak is problematic. We kind of average, while sperma one happening provides, which makes it machine unusable. I, see.

B

Okay, okay, so this actually is the okay, the kind of good the writable layers of a container like no part of the sandbox, but like some some some somewhere else in the class image right.

C

B

B

So none goes making. Ref has to really own it, so you guys cannot do that.

C

From my from experience right now, this will really limit. This will force us to reorder every every docker image we have and I say because, for example, if the user third is very often that example ideal, the docker image using the third-party library mm-hmm.

A

C

Already, the author you're, the creator of the image will not audit every lines of traffic and libraries say: where does it log I'll create a temporary file, although all these things ya.

D

C

That we don't know that either so and if we make this read only thing, we really feel late much later in the projects, I.

B

See, okay I see got it okay, so on K store down. If so, what what's the I guess.

C

We can't you can order to escape the background part yeah.

B

A

We're just signal.

B

On the background part.

A

Yeah so one quick question, one quick question: so the limp in the loop FS sighs here it means either when we eat permission the raw file system, it cannot exist limit of the user set or after the request, preparation finished. The container can now accept that limit. So I am still a little confused on this two part because I miss the Christopher Chris conversation I.

C

Am NOT very explicit on this part right now, because I think whatever the whatever the root has has side the users after the provisioning, but before we start a process that that typically is okay, like I haven't seen that causes trouble in any practical, practical experiment, because that's at most destr, that's not going to be bigger than the talker image size, which we typically try to build higher. At least there's they're pretty easy to know.

C

But after we get the image up and whatever the process doing from at this point is out of control of us. Okay,.

A

So the limit thing here it means like in a limit what the container processed right to the disk.

C

Yes, although, if we, if we're using copy that, because the the provisioning part also takes space so for other for other all the other three backhands, the provision itself does not create extra disks is during provisioning, their image itself occupied space, but there's a calculative separately, I guess: okay,.

B

Yeah so I think I feel, like we're part need to do this. You know a back-end specific way because I think for coffee back and you do need to limit the size for the entire provisioner through AFIS, because that's kind of private to each container well for overlay, the part is actually the upper layer, because the lower layer is actually sure you cannot like limit the size with that like how do we account for the size for that I? Don't know, maybe like it I, don't.

C

B

Maybe it's better to like also account for the lowest layer. I mean it doesn't make sense to you. um Yeah I, don't know. What's the best policy for overlay.

D

Because this is really like a shared.

B

Layers like like I think we can conservatively assume that container or executor has to declare resources for those space as well. I.

C

Know so we is interesting like we have ed over, we have two type of cluster our service cluster. The layer sharing is not aggressive like between different images. They share maybe I'm, making that making less than half of the layers. However, in our batch cluster we pretty much. We probably launch one docker image for every for all the tasks.

C

Okay, so if we another is a very big image, but there's only one copy on a machine mm-hmm if we charge this to the container the amount of disk the disk, a calculation is definitely that face seems too conservative, I.

B

See so you're saying like in some specific workload that calculation might be too conservative and doesn't make sense, correct, correct, okay, yeah I think we part need to design some Sun like policy there, like one thing like we can just kept the over the upper layer size less one policy, the ought. The other policy is like just kept. The entire thing depends on whether that the user wants to be conservative or not um and and then we should let the operator choose which one to use I, guess: yeah.

B

Okay, so do you have a kind of a design like? Oh, what's the kind of a design.

C

This just like to high-level directions. One the right direction is to provision this mutable past in the sandbox yeah.

B

C

Sandbox is the directory that I said all of these guys later it's closely tracking, that's provision, I think I believe if we simply provision all these mutable paths there we can, we can achieve something pretty fast by in the US and other the other. Both both discs release Isolators can can just track it outside and the report. It can also report I, think I believe we can be proud. Post members, as reports call the sandbox I than the root filesystem sighs. It's a fun member yeah.

B

So yeah this has okay, yeah I, understand this part, so um yeah so I think the only it might be complicated because the way we do by announce, whereas in sandbox, because we really like my mom, the sandbox itself to the real fast. So did you see I'm saying it's kind of a cyclic dependency yeah the sandbox depends on real fast, then I mean previously right now currently, so the sandbox does not depend on reference right.

B

C

B

Is does not depends on sandbox, oh sorry, the reverser FS does not depend on sandbox, but but your.

C

B

Done adding this dependency because the the upper layer is inside the sandbox, so the brew fic essentially depends on sandbox and semo's depends on reference. So there is a sacred energy so in mind not work. Yes,.

C

James James P she's, not here, but he actually suggested. We create a parallel structure.

D

C

That will change the change. The current level links a move, move the actual sandbox while level down and then create a parallel layer of the root part.

C

So so the sandbox in the root is so and I believe right now, the sandbox directory is something like slash framework IDs, execute, ID, slash their runs, and then the UID I forgot what our idea is yeah, so you are used to continue ID community. Then then we treat that container ID as a sandbox. Maybe we can probably do what level they'll say. One of the minutes actually create a slash, sandbox and a slash root. This is the hidden name he's picking it is, but having a UID part I see.

B

Yeah so so to complicate complication is you do have an image volume which I don't know? Who is using that? But we do support that that inside essential for each container, you might have multiple references.

C

Is that you that's also writable yeah.

B

So that rule F is writable, it's the same ufs as if it was the yes, so the last story for that image wrong. Is we used to plan to use that to support thermals because thermals wants to screw themselves, so they don't want to prepare their FS by themselves. The one makes us to prepare a reference for them and then they can screw into that red vest.

B

So so down with that's the reason we add this image bonding thing where we use the same provisioner to provision or ufs for the container, so there might be multiple image volumes, which is useful in general, I mean for debugging. Maybe who knows like, maybe in the future, it might be useful, pretty interesting potential.

C

Case because we have been heavily invest on docker image, distribution, yeah.

B

C

Thinking can we just wrap any arbitrary kind of binary data as one fat layer above a base image? Then we simply provision a new image volume. Every time we needed to do some kind of large data, yeah.

B

So that's one potential usage for that sure yeah yeah. It goes through the same provisioning steps that we saw for the Roo FS yeah, so so so so Tod are, for. This is having a root inside the the sandbox directory might not work because we do have the image volumes that will.

B

A

Know use spaces.

C

B

Image volumes there yeah, we can I think that's.

A

A big change, yeah.

B

Okay, maybe we should go through all that design, like all the alternatives. First before we kind of thinking which, which one is the best not understand that you want to have a I think this might be something we want to do, because I think this is just something that we regret, that we didn't do initially Syria.

C

B

A like a subdirectory for just sandbox in the in the in the worker for the container. Sorry for that for the executor right. So this is just something we regret that we haven't done, but right now, it's hard to change, I, guess and just fall backwards.

C

Concerned about the the migration over like the code needs to handle because the hot needs to handle both cases of how the sandbox is provisioned right.

B

Yeah, okay, so what's the alternative.

C

Mike I didn't play like I, haven't even really expand. How to do this, as providers probably means we need to send more information to other post, disc Isolators and say they I think right now the isolator only know the routes file system path. It does not really know anything about the backend or any which how this is done, but other religious.

B

Right but yeah I think we can get the back. I mean I, think we can get the Rueff s directory for each container right, mm-hmm, that's something we checkpoint already. Do we.

C

B

Part of the container config that you already checkpoint it, but that's the provision, but it's not the upper layer, I guess what you really wants to upper layer, another entire affair right.

C

Right well, I guess, if.

B

We do the policy that would be be super conservative, then it's okay. Then we don't have to get up. But if you won't just do that for upper layer, then you have to get somehow get the upper layer, but that's backhand specific knowledge right, I see yeah, that's the tricky part.

C

So I think I guess the question here is: do we want to open up these backhand specific information to persons yeah.

B

We can why not I think that's.

B

You mean backhand specific, oh yeah, it's it's more like a like. You know like which backhand you're using and then to some helper function in the backhand I can directly call you don't have to know the detail. You just need to know to call that function.

B

I may be generalizing interface, say hey. These are the disk space. Then these are the directory that um whose disk space might be counted towards the quota of the the container things like this and the ask-ask the back end to return a list of directory, or just wonder actually that you can actually do a deal in directory.

B

C

It's a later can query back hand I.

C

Think I think we, the new yeah I, think the the interface this part is sounds. A good idea is actually new. Is this ever? Is this ever mutable? After this information change after container comes up, no.

B

It's there right, so I, don't think it can change after yourself. So.

C

There is not going to be new past. This needs to be tracked after the Isolators called started.

B

C

B

Like the only thing that's there might be, mutable is volumes like persistent volumes that you can add after containers, launched I, think the tracking of the size of persistent longing is already handled by the Isolators and.

C

It should handle different. They should not be counted in this part anyway. Yeah.

B

So this is more like the sandbox usage right. It's like private usage for each container. It's not like volume, so the index is the image warning should. Should we do that separately or all right in which one is interesting? Yeah yeah I feel like implementation, wise, optional. A medium work is.

C

Actually, God Inc incorrect impression that immediate media volume was read-only until this meeting I used.

B

To saw the image volume border is it's not not read-only, you can actually write it.

C

B

Actually going to use it if they plan to but I, don't think they're using that right now, I, don't remember, because why don't they simply move to NASA container what they can adjust like the new work to make that happen, but it seems like amount of work at this point.

C

B

Okay, you screw up a little bit. I need to think of it on the first option. So James me just say: can you house container to right you get the disk resource applied, but continuous embodies the sandbox is to overlay layers, I see yeah. So this my work.

B

Yeah, so this is basically for the the upper layer, I guess.

C

We can even expand the selection route part to put all not only the roots of system and also the image volume. We can also put a the the middle part of it. You need volume there too, yeah.

B

So anything, that's private to the container should be put there. Yes, basically, yes,.

C

Yeah and I think this also this these are leaves leaves the further space up. If we do other things, we can just add one more one, more parallel level there for future things, because all the disk is tracked on the container ID part yeah.

B

So that's a fundamental change of the layer so like what about master container are you're going to be storing the sandbox and their root. Containers, sandbox or sandbox continue IV sandbox.

B

Where do we put right now? I surrender, there's only one sandbox for each container right so and we have a containers subdirectory under the sandbox and then container ID, and then that would be the sandbox of the nasty container I see so yeah, it's all merged into. So if we want to split these two, which I think it's we should do like just like backwards. Compatibility story is hard yeah. Let's say we have to. Then you have you can't have anything. That's user visible should be part of the sandbox.

B

They were actually anything that user should not see is part of the root directory right. You maintain this parallel structure for each container, even from the container right, which you can definitely do.

B

Yeah I think that's possible. Just backwards-compatibility story is gonna, be hard.

B

Yeah I think it's more doable for the second one is more doable I. Think the backwards-compatibility concern is I. Don't know how to solve that. Unless you just tell people to drain the entire box, give me all the satin boxes.

C

Okay, I think there's pretty much conclusive I did I didn't have any any write-up about a standalone container cause. When I wrote this, it has not been landed yet: okay, how how I stand alone container? Does it have a set box yeah.

B

It's it's the same as the same box directory is not in their slaves work. It's not under, like the traditional like sandbox for executors. It's just like top-level slave work, their containers and then continue on D.

C

So that's the sandbox and then right now the root filesystem is still within in the provisioner yeah.

B

So right now all the real effects, including the upper layer, is part of the provisioner.

B

Yes, so I think one thing that like, if we don't, if you don't want to do this, like a per pack hand thing like you only do that I say just just to make sure the entire Rufus is under karate cows. You can just run D you on the rough s directly.

C

We also, we also want to move away from the. We also are looking at moving away from to you slowly into either XSS all right up, maybe put the project code I, think on to the exit e4 when we upgraded the latest Colonel Dan.

B

Even is there you can like, where year x8 like XFS, you can just use like a sign the same project ID for the upper layer. Tutors just use the same project ID. Then it will be subject to the same color. Well,.

C

Yeah the reason the reason is like tu does not. The reason we want to do that is to you has the limitation that it does not know it does not track styles, which are deleted, are still being held, but still has it handle being held by something like if I have a files that deleted, but there's still another open tour descriptor.

C

You don't see it I, see.

B

Yeah yeah I do yes. Ideally we should purchase use XFS or some other ways to make that so that means like we only have one option just put that well. Otherwise you have to make sure that you can understand the structure of the the provisioner yeah I. Think the long turn to rush I really like like prefer the design is actually something you want to do long term. If we could have a backward compatibility story: okay, yeah! Maybe we just spend some time thinking about how to do that. That's something worth thinking about it!

B

How to do that?

B

C

B

What's the timeline for this well.

C

I guess this is not going to happen 1.5, so maybe I want to target to say if we can have a some mechanical female point. 6 mm-hmm.

B

Okay, I see okay, yeah. Okay, we can yeah art need to think about that a little bit more and I think we should spend some time thinking about the preferred design, because I don't I'm, pretty sure how to do that for the alternative design you just pretty no pretty hacky, yeah I. Think the rule of thumb here is like anything, that's private to the Container. It should be part of the containers, ephemeral disk space and that disk space is part of one single directory. So that's much easier to track.

B

Yeah I, think that makes sense.

C

Okay concludes my my part. All.

B

Right so I think I don't have anything else for this working group. I want to have a demo for standalone container, but I think we're too busy that we don't have time to prepare for that. So maybe next time and and yeah I think things like I think like next time, I'm gonna send out email early and then we need to go through the the priority list of things that we want to do.

B

I think like right now, a lot of thing are finished and people are working on something new and it's it's time to update that spreadsheet and see what people are working on right now and then what's the plan for the next six months, things like this all right, I, don't have anything else.

C

I, don't have a cup like a couple of quick questions, especially afternoon standalone container. The first days I think we still plan to build some introspection API to list all these containers and other configurations right. Yeah yeah is that is that plan to be in 1.5 mi6 or somewhere yeah.

B

So that's that's what we plan to do for I hope to get that in one file. Okay,.

C

B

Right get I mean we're basically argument to get continuous operating API to allow our priority list, not just the the normal containers, but also the standard container, as well as nested containers, because right now the gantt containers only show the top-level executor containers. Yes can.

C

You CC me on those Patriots went ahead: robe yeah.

B

Sure, of course, yeah I'll create a key I, don't know their superiors to keep with that so Cobra I, remember, there's a ticket from that showing nasty containers uh also remember: there's a ticket. Okay, I, probably.

A

Requested that yeah.

B

Yeah yeah, we should definitely do that and we are. We pro just allow people to specify some parameter in the Opera. The API say: hey, show all the containers or just show nasty container. Just basically I feel filtering stuff. Yeah.

A

After the checkpoint, anything it should not be hurt, but yeah.

B

I think even before check Pony, it's not hard.

B

Anyway, so um yeah I think we could do that. It's not that hard and just adding some implementation there yeah that's a plan, that's the Poncho's I think what we do right now is that we kind of come for the stand on continuous stuff. We kind of combine like so so. The launched nested container on API will be deprecated in favor of just launch container, which can launch both yes ticket container or top-level. Can you hate?

B

So that's the change we did and the Oh API steelworks is being deprecated, and then we use the continue idea to tell that it is a top-level stand on container on nasty container.

C

Another question is about the security like, especially for the default installation: I, don't know how many people actually set up the entire credential ACL on our agent side. So this is actually not super tribute, not super truly to set up, because in a class at her nerve agent is almost always the first element you get up and the dependency of the agent to other security or secret distribution system is often tricky. So obviously we don't have any secrets or credential set up an agent side. Yeah.

B

C

This kind of do.

B

You see less yeah. The u.s. is likely to have a centralized server key management server that you can hit on to to do authorization authentication, but but the default one is actually based on some fire on the localhost right. So that's not on well I. Think I. Remember you! We used to do like just have puppy or chef to populate those keys and then make sure they are protected by root user yeah.

C

So I think the reason I bring this up is after NASA container and it standalone container. There is actually a extra attacking vector in the agent part before these two API is actually people cannot mutate. The state of the agent through a 1080i I believe these are the two. This study.

B

We add authentication authorization to yes,.

C

But I guess for default installation of a methods agent which people don't set up credentials or ECL yeah.

B

But you can set up a cos. You just disallow any user. You.

C

B

That you mean that's just an agent flag right. Oh so,.

C

You mean people can just simply set how the ACL and say and any image to launched. Within these token here, yeah.

B

Only allow one particular user through to call those api's principle, but you hope you have to set up both authentication and authorization for set up authentication right now. I think you can use the basic of, but that means you have to save some credential file on your on the agent note, um which is a little tricky. I have to distribute a key to every single note row taking it actually, a mop is a bigger problem and key rotation is hard.

B

Yes, yeah I, don't know, what's the better solution for that, but we do add, authentication authorization for that, so I.

C

Mean if we look at other system like what docker do is doing, is that the default, the only this part of API, only listens at a root you curve at the root own the domain socket. So we saw the call the caller part needs to have. It could have been a good permission to actually use the code, easy guys, yeah.

B

But that does not work right. That means, like everyone wants to use. The API has to have real privilege, which just doesn't hurt, doesn't make sense. Oh, we.

C

Can explore me we can definitely expand, is I'm just saying that's the default behavior of the talker, like the wouldn't. Would you get out of a box when you install this without extra integration.

B

Right um yeah, so that's one good point. It seems.

C

A little bit safe slightly safer that what word do we yeah.

B

I don't know: what's the password I think Craig is on the line, but I don't know. What's the best way to solve that yeah.

D

I mean I I personally, I, don't think I care for the idea of protecting that API via file system. Permissions seems better to me to have a consistent Authority for authorization.

D

You know you guys.

B

Can write a module if you want to yeah.

D

I think you know the default implementations in Mesa, sir pretty simple, but but the existing authorization should allow you to to wall off. Any ap is or restrict them to particular users. Yeah.

B

I think authorization is okay, because that's just some config file I think like the credential part, is hard, but maybe you can you guys can use a custom authentication like that hook up without that or kirbo senior by.

A

Writing mojo so.

B

That every car is authenticated, whether it's an authorizer or not like depends on the flag and the flag is easy.

C

I think we are going. We probably will do that some sometime in a short future I just feel like we probably want to enhance it a little bit for the default installation, medals, yeah.

B

So that I can tell you what we're doing Twitter I will just use a happy to to distribute those keys. Yeah.

C

We are doing the same thing is just we are having how the requirement of not depending on another system and bring our bezels agent. Oh you.

B

Mean puppet, but we need to have happened.

C

But we cannot we the dependency to us. A secret store is very unclear right now, I see because of a cluster turn our process right.

B

C

D

So she charges stick clarify. Does that mean the difficulty is that here it's not clear that chef would have access to the relevant keys to the relevant credentials that you need? Yes, okay, actually.

B

Yeah, that's sure, to write a module. If you want that I think that's the only way I can think of right now, because you have special requirement, I think most people I put default. They don't care. People can not log into agent boxes anyway and then we'll be firewall to your given I, don't know what what can we do in firewall to prevent people directing agent API and.

D

Yeah, that's that's on a per endpoint basis. If you have a single endpoint, you want a wall up, that'll work, but.

B

Can we do that for a single I'll pray, the NPI I do want people to get flags, but I don't want people to get like the launched. Nancy continue our launch container no.

D

The firewall flag, in particular doesn't get that granular I see.

B

Yeah maybe like this is something we could do that have some that is disabled as an endpoint on per call type rather than per endpoint. We.

D

Could but I think my question then, would be why don't you just set a strict ACL that doesn't allow any user to perform that operation I.

B

Hate just ACL didn't outright I think the tricky part is standing, not the authorization part that should be pretty straightforward to set up yep.

D

Yeah but I guess, if you, if you're saying you want a particular a particular operation within an API to be walled off, and then that implies that you want people to be able to authenticate like walling off that just setting an ACL is sufficient to prevent anyone from reassuring that no.

B

I think you need on occasion right. Otherwise, I can just pretend to be root and I can do everything always wants to do or yeah.

D

B

Acl just allow anyone can use that end. Point.

D

No not the end point, but the operation.

C

Oh yeah, you could just allow.

D

All you users from performing a particular action I, see so.

B

That's just disabled that API right, so just how you can do that you can! Just if you don't use that, then you can just disable that API, but I think the executor wants to use that. That's the tricky part I guess we have yeah.

A

B

Maybe just bypass this is that correct, so I think the implicit authorization will like override the default ACLs.

D

Yes, yeah uh-huh yeah, that's.

B

Here execute can still launch that's the container, but any operator cannot do that. Yeah.

D

Okay yeah, if you're, if you're planning on going down that road using the ACL jito, then we might want to look at the code and just confirm that the implicit executor off is gonna work for you there, but I think that would do the trick. Okay,.

B

Yeah you can set up an ACL to just basically DISA just allow anyone to hit the endpoint that particularly launched continual launch the container, yeah I. Think that's one thing you can definitely do to prevent anyone like accidentally hitting the API, but that also means the operator cannot use that API.

C

B

And so that's just one safety net Dell. To fix this. You set up the authentication module properly so that I can set up the RAC. I'll say: hey. The only routes user can use that. Otherwise anyone else cannot use that.

C

B

Right, alright sounds good um thanks. Cameron I think you guys seem to weeks. Okay later.

D

B