Apache Mesos Containerization Working Group, 8 Mar 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GMT 2018-03-08 Containerization WG

Description

Agenda and notes:
https://docs.google.com/a/mesosphere.io/document/d/1z55a7tLZFoRWVuUxz1FZwgxkHeugtc2nHR89skFXSpU/edit?usp=drive_web

A

What to do yeah you, so you do like.

B

B

C

That's wait. Wait for one more million that guy started. Gilbert. Do you have anything to to present for the first item.

A

I I might share my screen.

A

A

C

All right, I think, let's get started Gilbert. You want to share your screen, I'm just unsure my.

A

A

A

Okay, I find a button.

A

Hey morning guys so, okay guys, let's see my screen yeah, okay, so yeah. So for the for the first. What first agenda item it's about the Daugherty man handing issue? So if you would be pretty quick like ten minutes and then we hand it over to town, so basically we have the dog a demon. This is so we fix and epic for the doctor demon hanging and it is not back on missus. It is a back on dog, a demon so in production.

A

So if the dog, a demon like there are too many come and go to dr., COI and son come in my hands forever, and this is a big impact on / because it will result in many different.

A

Symptoms, for example, tasks could not gonna, be kill forever and missus. You never know a task is running or not so. Missus have no idea what the task status and eventually it turns out to be their capo praise. We have to work around on dog a demon to do with the Double Diamond heading, because this behavior is not consistent. For example, sometimes dog the whole dog, a demon hands. So if you hands forever no matter, docker PS don't run any dog a comment.

A

It cannot be invoked being executed and or they're, not a saint, and it is some of the dog a comment. It does not work and hence forever, but if you invoke the second time, maybe work, for example, nothing instead. The first time is just hanging there and if you involve a couple artists, some of them Isis something might not and there's ammo be issue. An open, Lobby issue on the mobile repo and I think it is. It was placed submitting in under this epoch, but they did not fix it yet.

A

So we have to work around this and I believe the we have this epic back ported to 1.5 1.0. Once three, these new versions and I breathe. It will improve the user experience quickly for people who use docker demon, introduction. Okay, let's take a look and what exactly they are so the first one it is the one from fixed by Chen and basically this is the issue.

A

This is the issue about when a docker container is already running and talking and if in it once they finish or exit, the other demon could not capture the access status of this container because of the demon issue and I think he laying a ticket on the mobi repo.

A

So, basically, basically once the docker demon could not capture like the container is finished, so Mesa spirit card container still running, we have no way to do the status updates, the scattering, and instead we implement and walk around. So we use the link next rip to rip the pain of the docker container. So once we have the access status, we just return corresponding test status like test finish. A pass failed to make those scheduled, and then we will killed you.

A

We will not rely on any any other information from the demon so because it's create this, what taka demon do because in double-teaming it also rely on Linux content to read the container pit. So so because before this fix, we have a very bad situation. If the doctor didn't hand on that, and then, if our scheduled retry curing this test, it could not be killed because or the behavior rely on target even or the continuous are actually cute or finished that we just don't get the status update.

A

So this is a workaround we is already by party I. Should I should just the fixed version day, then it should be now at from one party to the latest version. Just a time tire. You have five more minutes: oh okay, okay, so the other one. It is the about the fixed we do together from grad myself and entry, and maybe I can I could introduce them all together. So darker demon could hand in any comments and they just rely on token inspect, to get the handle of the continual pet.

A

So every time we want to get a handle of this container, we need to call doctor inspect and doctor in Spanish was caught in many place. For example, when we pull the image in dr. continue noise that before pulling Nico Douga since we're in toka comment as cutie, we call doctor instead after daughter run. So if any of this doctor is the hand we have no way to get the handle, so we could not do anything if this docker container.

A

So, even even if we call this car, we don't have any other.

A

We don't have any other update with this docker in Spencer process, so we update the we change the library it's just done by crack, so we support the disk, our behavior from, for example, if the color of this docker library called open. Instead, we call this car so in the live. We have a combat on this guard to handle, like all this promise, for the method of a stop talk abou order. Instead, we will make a discarded future and return so, which means we will not have a pending future forever.

A

We will return the discarded if the caller discard caught this car. It is a litmus test, is kind of pretty early process. Things might be confusing, and so we update the library and then in the continuation and as a killer, we put correspond a fixed after docker run and dr. Khoo. So every time we after a timeout, we will discard this future and then we will return either failed status or discarded status for this particular. Instead, don't stop or don't approve. So we will not end up with the situation in the continuous daylight.

A

We are launcher continue and then it just handing forever there. So from the sake from the user side, user could always get best something for the content from the continuous status. This is another thing, and the last thing we improve is from NJ it's a film entry, so we fixed the. We fix the darker common as a cutie.

A

Also, we add some improvement. It is the rich high logic, so we implement some loop in the Kotoko. Continue nicer and the docker comment executed for the docker inspect. So let's say: if the token is the hands, we will retry it again and this which I can be forever and.

A

A

Basically, the retry, it does not change the retry logic of tokens back in the command executed. It does not change any semantics before it just improved in this improve the experience. So, for example, like the first stop into the hands, we will retry again and it's very likely for the issue if the daughter demon hands for a couple tight. If you adjust this issue, okay, I think that say so.

C

Any questions can you adjust the fixed version to properly reflect which pass you have backported yeah.

A

Yeah I just merged the I, just enriched or I just spread for all the patches yesterday, I just I. Don't that just don't have the time to update the fixed version. I will update and today, all together, yeah.

C

Yeah make sure to do that as I do a show up in the change lock, yeah.

C

Copy the changelog to you I did not, but it is on my. It is on my local authority back for the patch, but you haven't had.

A

To change log yet I, because the change log there, some country, I'm, still fixing so okay.

C

Yeah, so you'll have a chance to push it. Okay, I see yeah, make sure to do that. Usually we do the backcourt, along with the change log update. Otherwise it would be very hard to track those back ports, yeah yeah. It is already.

A

On my local pet, I just didn't appreciate it. Okay,.

B

Yes, oh, this is great. We actually see quite some darker and stability now cluster to the degree that other words of giving up this thing so I think I. Think this case can increase the quality of our what we offer to our internal customers in this you know in a short future, a couple question is number one. These improvements are people to both a talker executor and the custom execute are using docker container either.

A

Yes, yes, one of them might not be there yet so might be. The might be the first one takes my chain, so this one I I realized the fixes for the executor only. We should do the same thing for the custom executed, but there are all the other things they are fix on the library site or the continuous I, so they should be sufficient for the person executed as well. It might.

B

Be good to so, secondly, is like I: don't think we I don't think we have time to go over the hot Holocene, somewhat a symptom, it would have sake fix, but I think it might be a good idea to call out what kind of symptom people typically reserved for the bug before other fix and after the fix. So they have a better, better idea to understand whether this bug has a good chance to fix the problem. Yeah.

A

Yeah combination.

B

Between agent and the doc remains quite complex, various combinations to the degree that few people in the world, especially outside vessels that team actually understands how to deal with all of them. Yeah.

A

Yeah I actually I have a user experience talk on what symptom if they are, but they still contending that they know that.

B

Somehow that will be awesome, yeah.

A

Yeah, yes, so I just before the meeting I just didn't, don't have a chance to fight it. It's hiding somewhere one-page talk, the Google Doc, so I will I will try to find it out and paste the link over.

B

C

Think may link the talk to to this epic or any of the tickets will be useful for all.

B

Maybe just convert that into a I, don't know whether that should be a documentation. The metal itself has a bug fixes in our own doc.

C

We don't have that precedent in.

B

The blog very good I think.

C

A

I think a blog will be great. Yeah I agree: yeah I. Let me think about that. I think I was I agree, that's a turkey. It was a Brock, but for the I think for this ticket we each take it. We identified the root cause of each for each symptom. We just need another Google Doc paste on this epic I think go to let people know what's the symptom of from the user side from the user.

A

What will user could see and must use the quiz suffered and and if there's a chance- and if we have time I would yeah I will happy happy to help on writing a proper host about like how we improve the quality? How we work around the dr. Beeman issue on basis, yep.

B

C

Okay, so so just have one more question, so are you guys using docker container with custom executors? Yes, oh I see it got it so so the executors were any inside docker container. Yes,.

B

Actually, we have yes and that's that and we also have a let another, pretty big, a cluster running, docker, executors, okay,.

C

B

Almost every combination possible: you have a clan so right now: okay,.

C

I see got it okay, because I think that part is not well tested, at least like we don't have any unit tests for that the custom and secure, because I don't think we have any well. We used to have one but I. Think it's not actually I mean it's the coat color. It is pretty low on that code path, yeah.

B

I think if you I can think of ways to improve the test car radio. So if you tell tasks- and you can almost I- might need them direct to me, I can't figure out a way to improve test coverage on them. I'll find someone else here who's over here we have a couple engineer: increasing contributing to metals. That's a great start, starting point. In my opinion, okay.

C

That sounds great yeah, I. Think um yeah, definitely shout to me and I can point the way of doing that.

C

Okay, what's next shiitake, you want to talk about the the meters for nine six, four, nine, six, five.

B

Yes, so so this is so this epic is about this issue is about allowing the sides of persons volumes so I think. Do you want me to protect the talk from my side, yeah.

C

I think that would be great, so go became unsure and then let you talk share his current screen.

A

Yeah I think to tell you, kids, sharing it yeah.

B

Can people see the talk yeah.

B

So I think that so this is just about surprising, resizing present volume, so this economy has been there for I, think more than two years and uber is actually running a very big storage cluster. Using this feature, I think I think we are running one of the largest Cassandra installations purely on this class right now, and it's recently brought to our team's attention net because we do not have. We cannot resize presence podium.

B

What ended up from storage team is they had they allocated a very large volume size for every customer in, but in practice wasted a lot of disk space. Because of that and to arrange these things around is very operationally extremely efficiently expensive and then they don't want to suffer the down times. They move to a smaller volume to other things that both make their framework complex as long as as well as probably possibly bring down time to the customer, so they were really requesting.

B

Can Nestle support recycling of these volumes as well as the second sense is safe and I think this is something worth discussing, because this is actually translated to a lot of cost-saving in our organization.

C

Okay, so I'm just trying to understand so by resizing. So, first of all, are you guys using Mount Warning or we.

B

Are using we are using both, but I think the parts they need is only the path body.

B

We are simply creating pretty volume on the root disk. Okay,.

C

And okay I see, and then you want to resize that I'm persistent long and so that well I think if it's sharing re-enforcing disk Hoda I mean if okay I see so I see. So the reason you want to do resizing is because you want to make sure that the colon enforcement is working properly, because otherwise they share the same disk. They can just easily go up their shares, but even that we create problem allocation because they.

B

Are relying on the same yeah right.

C

Okay, I see got it okay, so.

B

I think from our side, the requirements, especially for the first phase of the project, the the the necessary requirement, is we want to be able to decrease the size of a persistent volume so in the free up the disk resource so other we can use for other purposes, I think if we introduce this API for the sake of usability, we also want to be able to increase the size so that, if something, if the adjustment goes wrong and people realize they pushed too far, we can trivially increase the size a little bit as long as there are still disk space available on that machine.

B

A

B

Think in MVP phase we can totally accept that both increase and decrease can only happen when the volume is not being used by any task or executor. I mean one is offline. I think it simplifies a lot of problems and, like I explained from our side as well as I believe he covers the route tap disk. We will be happy I think the work should be easily extendable to path disk. If my understanding is right, don't defer that much mm-hmm.

A

B

The mounted disk resizing seems quite tricky because, actually very, like is a different thing in the mountable, so we are not active seeking for that, but if you think people think this should be including the apartment we can revisit well,.

C

Yeah I think mount is the toe depends on the owned. At the back hand, I, for example, decreasing, is most likely not possible in many of the file systems. Like you know, you cannot simply just say: hey I'm gonna remove some of the disk space. From that virtual disk, like I, mean the expanding is possible with something like LVM I can't increase the size all by we're logical desk and then that can just essentially increase the size of her system. Long but decreasing is gonna, be harmful. Now, this okay, let.

B

Me let me say this is all side of MVP scope. At least yeah.

C

B

So so I just so I started a quick, quick document, quick working price, the dog could capture, seems like what's needed across the stack to make this happen. I think the first thing I want to discuss here is the weather. Api should look like mm-hm. It seems like the the seems, like the reservation reserve and I reserve. Api right now is incremental.

B

So, for example, what I mean is if we have a one gigabyte of disk reserved on this machine and I want to make the total reservation to to think about. As another reservation, one gigabyte mmm-hmm instead of say I was a total reservation to two gigabyte here. So that is why I list how to alternative the api's ITR direction, because I will be describing a grow or shrink of the doubt apart and the other would simply be update message describing the total size of the volume right.

C

So yeah I was thinking about similar, like I. Think I was thinking just like adding a new framework offer operation, maybe like resize volume, I agree of onion or shrink volume and that operation you should take the original volume and plus some additional discs resource where the additional volume size is gonna, be coming from yeah and they have to be reserved on the same row. Otherwise master will just reject oasiz. Yes,.

B

That's exactly what I'm describing in Africa, right and I think I. Similarly, we'll just add a similar things to post the operator, API and HTTP endpoint, but we still do bad produce changes with HTTP endpoint.

C

What do you mean by HTTP endpoint, oh.

B

C

Oh yeah, we stopped doing like v1 sorry, v-0, I preview, one out for the api's, so you're probably gonna cross this out, but I think it's.

B

Tribute to add it if you want, but yeah.

C

I mean we try to not to undo any change to the v-0 and encourage people to use. We want okay, so I.

B

Think information wise this should not be hard, especially after recent changes, post CSI and it's a generic configuration, because research has already HK a lot of resources can really change. I took a quick look at the code that understand was necessary seems like only she know. It hinges on alien science seems virtual, almost a trivial in that master simply needs which has send a new checkpoint, the resources message and let agent apply it if we and if we believe the the volume is offline at that point, and everything is fine.

A

D

C

The I think implementation was should be, should not be too hard once we figure out the API I mean there's some. There might be some tricky kids like what if the RO doesn't match something like this I think.

B

It's like more work will be all the validation inside inside right.

C

B

We also I think I think a tricky corner case is if we decrease the modern sites through the usage like, for example, if the database left King gigabyte of data on the volume, but we are shrinking it to five gigabyte. What does that really mean yeah.

C

So that's the tricky like one. You guys need to shrink the size of per system on Ian like like.

B

I said, for example, our Cassandra team I think we gave out because, although.

E

B

See I think that this base is when the long to the product. Now it's very hard to gauge that so the case a give everybody a two to five, a terabyte of volume space, but then a lot of product and not smaller at the end the day, so they say they probably need, maybe a hunger, a 200 gigabyte in stable state Sandra has the behavior. They need double the disk space force, natural, absolute necessarily needing is less than 500 gigabyte.

B

So they say they want elongate, let this table and then say: okay, let's now shrink back the space, so we can free up or other needed. Okay.

C

Got it okay, yeah I. Think, as you mentioned here, I think. One issue that I can think of is I. Think for D you base isolator. It should be okay, because once you yeah I should be okay, you should adapt to the new resources, but I think for XFS arm-based eiseley. Don't need to be extra, careful, I'm, not sure if that's supported right now,.

B

Someone someone just caught it out yesterday for for me, oh.

C

B

C

So purses ammonia is not supportive for XFS. Okay, I see you got it.

B

And I think as long as this is offline, it should still sugar still work. Yeah.

C

I think right now, I think the file the problem, because you require an offer to be able to do resize. That means the pony is not being used by anyone when you drink, grow or shrink. So I think that simplify the problem.

C

F

So one common here is them are like so some of the like. We probably should think about adding the same ability for, like you know, a capability for a shrink, and that's also I'm sorry, so I've got adding a capability for shrinking so which would be you know, determined determined by the actual implementation from the precision volume. Would that be a consideration?

F

C

I think like moving for so this is just for agent, the resources disk resources provided by agent directly I think, given that CSS being introduced- and we do have CSS support for local volume, I can read something like LVM right now so moving forward. If you want to do resize for a CSI back to volume there, the CSI plug-in should give you information about whether it supports resize or not, for example, I all VM support resize, so it should support at grow.

C

At the very least shrinking I, don't know, maybe it doesn't support, but yeah I think we part need to discuss with CSI community on that to see if they're interested, adding a CSI interface code, I can resize I'm, pretty sure you're interested, because I see discussion on that previously and and to make sure that the CSI back to volume is also supporting those new offer operation like shrink, otro right.

F

And adding like affirmative or you know, our capability of Jana I think this would be a helpful to identify. Would.

C

F

One of one will do that or not yeah.

C

It's up to ya. Ultimately, it's up to the backend to decide whether they have the capability.

E

C

Do shrinking or a grill and and I think the question is: how do we expose that information to the framework? What kind of API we exposed to the framework about that capability right now? We don't have that mechanism going to think about that, but at the very least you can reject. If you don't have that capability, mm-hmm.

E

C

B

So a question about a hell of a question between prison volume and CSI. There is no current plan about the changing of the volume to be back from the CI Sai interface. Yet right so sorry, City, yes to talk so so present modem is, will still have their own API in the short to mid-term future. It's not.

C

B

Require to go through the series, I know.

C

I mean these the persistent morning, API is totally out out to the tutor like Trayvon in the live audience years later. The API, because once you create see us I mean the CSM only just like create voting from a poor storage and then on whether you want to use that voting as an ephemeral volume or pursuit ammonia. It's another decision you had to make. So these two are fully open. Oh I see okay.

B

C

Yeah yeah, we cannot hear you oh yeah,.

D

C

D

So, just to think so, I was trying to chime in when we were talking about. Even for this to you about downsizing and and the size of the data on the disk is already larger than the size you are trying to shrink to.

D

Even if you have to you, you cannot prevent other tasks, other containers from being prematurely limited because of the shared disk right, because if you downsize your disk and you're, not launching a container but someone else thinking they have a larger space, but then they could run into a situation where they actually don't have that space.

D

You already talking about share resources like so, let's say: if we're on the path disk mm-hmm, you you have to pack disks, sharing the same physical volume yeah and you shrink your bowling one, but you've left more data on the disk. Then you're, a persistent volume uh-huh, then you're affecting.

D

The persistent volume 2, which is on the same physical disk, so then another tasks using the other volume cannot actually write to the same amount that it's offered to it right. Yeah.

C

I think we I mean my understanding. Is we already have that issue even right? Now we don't have any grow or shrink capability. We still have that issue right now, given that the D you based isolator, will still allow the disk usage of a container to go over quota you're.

D

Saying it's not, it was not in enforcement or eeling.

C

Enforcement, it's possible that the container that um that I used that the persistent moment will go over code that because the U is based on time and the time you do, the check is, if it later than when you write a disk and you might still go over Cola and your container will be kill eventually, but that the disk space you're using for that container is still go over its quota. So that will affect other containers quota. Well, this that.

D

Will be basically as a result of a race condition if you will, if it happens during the the period yeah, but this could be a long time right if, if the framework just stopped launching any chances and goes away, it will be basically staying that way. Yeah.

C

I think ultimately, I think that the past base disk is very hard to I mean unless you use something like extra fast to to enforce strictly enforced quota. I, don't know like, even if, in this case like like how do you reject the shrinking if the disk space used by that contain like that continue on that particular bond is already go over coded I wish. Should we do so that there.

D

Might be some teams.

C

Just think about well.

D

If you have explicit feedback for the current set of operations, then you could you can send that back as a failure to.

C

To downsize right yeah, but that that means agent needs to get the information of the usage of that disk somehow in some way right now we don't have that mechanism. Oh yeah, that.

D

C

A do you when it's doing this operation and yeah, we can do that. Just okay, yeah I, think that might be one possibility to just do a deal when you actually do a shrinking or just additional checks and recheck the operation here. If it's not under coda yeah.

D

C

That's a good point. Didn't.

D

You think about n right and X of s. It's it's not in it right now, but logically it sounds. It will be only natural to support it. Yeah, but I think it's it's not that different than supporting all of the cases we'll have to consider for this to you.

C

D

Yeah, that's a good point so also so for the CSI backed operations.

D

So, if I understand correctly, they yeah, like you, said it's its orthogonal.

D

But it's it's really in the sense that I guess cuz the they perform operations on like the raw resources which persistent volumes are created and top up right, mm-hmm. So does it it kind of means. It feels to me kind of means. You have the possibility of like doing to kind of resize to achieve one purpose like if you, if you create your persistent volume on top of LVM, and then you have to issue two operations.

D

First, you have to increase the size of the LVM volume and then another operation to increase the size of the persistent volume. On top of that, I will.

C

Be in Bulleid yeah.

D

C

One way to think about that by law, right, I, think about that as just one operation to expand the volume I mean persistent volume to me, just like a additional ID information and label on top of the using volume.

C

So what you're really talking about is to the one that has the storage is actually the underlying volume and the year that the shrinking and the grill is actually for the underlying bond and persistent money and just think about that as a label on top of that volume that also the framer can get that information and points the same volume when the task being restarted. If you think about that way, there's only one backing volume. So there is really one grille or shrink operation on that volume. Well,.

D

Can you have to persistent volumes on top of the on top of the same.

C

Lbm building can you have two-person Simoni on or not the mount phone yeah? So if you expose the LVM as a mountain volume, then you don't have a way to do that. Okay, you can also expose that as a path volume, you can't expose that as a path volume Dania centuries. Oh I see you're, saying like a you for path, phone iam that, since its shared, then you have another layer. On top of that, I see sounds.

D

Like many cases and possibilities, yeah.

C

So past funny, but I think it's it's related to whether it's past or my it doesn't. It has nothing to do whether it's persistent or not.

F

Intuitive really seems so I come out, not so safe on for my point of view, but, like my you know, I'm not sure, like the long-term, we should also do detect like the cardinality for this kind of things, reality carnality. So, for example, like my LVM being used as opposed to purses in volume, it then again it is being used them as a path line and it's being shared.

C

Well, right now, it's up to the frame to decide whether you want to carve a LVM boning group into either mount or path falling. So that's the frame to decide depends on whether they want to share this to space from that virtual, logical disk or not by multiple volumes. Yeah.

F

But like them uh again so like this would lead to like I'm, some sort of um you know, uh conflict or like a reuse of those volumes. Them are not in really not not really like in the desirable state right.

C

Sorry I don't follow um so.

F

I like come on, you know again, so it's just like I'm a year sharing the same route, these real disk, sharing that like the same quota and then there could be raised and they're not causing one unexpected up. Behavior uh-huh.

A

F

So I mean just on throwing of a point here: yeah yeah, so I mean not, like god, I'm, not I'm, just not sure like among going forward for a longer term. This, like cardinality detection, would be a point. Okay,.

C

I see: okay, okay,.

B

Yeah I will rephrase this in a different way, so, especially in a very complex crafter, when there's multiple frameworks and multiple workloads sharing, while really large results pool, which is the intention of how missiles to the work uh-huh. So what we vary in practice is that I think it's practically possible for every firm work to do the right thing, but one you mean: go code from different or written by different organizations or different open source projects versus closed and the close out project.

B

You don't always end it out all compatible with each other on the framework level so like in many cases, I feel it might be better as the operator to make three arrangements for these things. haha They simply say now. When you launch our framework, you don't really need to do any reserve volume things. Volume should be there ready for you, if not instead of directly interacting with a metal Slayer interesting with a very shape, very thin, shame of API server, which will make all these things available for you, yeah.

C

So that's what we're doing this us, so we in this us we have this centralized controller to actually responsible for creating those who volume and then we introduced this concept called profile which is essentially similar to like Ruby, nice storage class. You can distinguish multiple discs based on their attributes and parameters and the friendly one will just pick those tips based on the profile name of those disks yeah. So that's what we're doing these to us and there's a centralized component actually responsible for creating those volumes. So Freimuth don't do anything. I think.

B

C

Think that seems the we're.

B

Not us no, but that, like the.

C

Yeah, I think that you guys can do the same thing. Just have one another framework that that's as a central point of creating those volume for other frameworks, and then there there can be COI interact with that controller framework to create persistent audience.

B

E

E

To address question about increasing size, I think we can unify the is to layer, the backend volume and the persistent volumes of that by because we always need to first calculate so when we receive receive resource. Maybe it has a path online and we want to calculate the number of disk, not disused, but we so long to compute.

E

Before we also can long, you can compute the number of available space on the same back end source, so we and when we try to expand bodies, we proc we can first try to utilize the free space on the same bag and source. Then, if there is enough space, then we don't need to query the back end.

E

60S I began to expand the volume the physical volume, but and but if there's not, we use up all the free space, then we go back and query the the physical bag and do states to ask if I can do expansion on the actual baguette bagging how's. That yeah.

C

So yeah I think we should definitely do some design, because I think this is pretty early stage right now and there's multiple ways to do that so yeah, what you say is one way I think the other way is just like. Don't ask I mean there's some other ways to do that, where you always model the free disk resource as resources, and then we do crave on you have to specify which resource which additional this resource, this new disk space is really is going to be coming from.

C

You have to do that in the operation itself.

C

Okay, anyway, I think we should carry. We should not discuss the design here. I think we should follow up with another design, so this is a good start. So it's just ah what's the timeline I mean.

B

Ideally, we want to see this happen as early as possible, probably targeting 1.6. If that is technically possible, do.

C

You have do you have engineers that can work on? Yes,.

B

I will work on this. Okay.

C

B

This is top a priority in my development queue, so I thought that I'd find a Shepherd and the other change I will. This is my property about yeah.

C

I think I think June I think since you're quite familiar with step part. So if that's something that you can't help with gee-tar and you guys can design together on this yeah sure, okay sounds great. Yeah I think the follow-up binding. Is you guys just think on this and then come up with design dog so that we can discuss again in during the working group meeting in the follow in the next month or next two weeks, so that we can decide what to do next? Okay, so I think just a time chart.

C

We still have one more agenda item, so I do want to get to dance. So, let's go go! Go to tab discussion, okay, stop sharing! Thank you!

C

Okay! So um let me see the skin. How old are you there? I'm.

G

C

Okay, you want to share your screen to discuss the design doc for the excess salt and let me just pull the dog first.

G

Okay, so everybody can see this I'm assuming yep, okay, excellent, all right. So, just to give you a little bit of background.

G

Previously we were using the misty, you isolator, that we were talking about previously the problem that we have, and it looks like others- do as well tickets mentioned here, but basically there is a look when the hard limit is set. The current functionality of the disc FS is that when you set a limit, it understandably sets a hard limit, and so the result of that is just that. Your application will try to allocate space and most applications, don't actually handle the end of quota error correctly.

G

So they just sit there and you know, do nothing, do nothing useful. They may be complaining, but because you're in this space. Well, you you don't really know so, there's a couple ideas that we have and it starts out with providing a sort of headroom so and with this head room, essentially whatever the user.

G

So if somebody is using marathon- which is the case that we have what you would do is just say, whatever amount people request make the soft limit be that requested amount and then provide an offset such that such that it can be. You know, at a global level, you can say: okay. Well, we want to provide. You know it can be something as small as like one Meg. It can be 10 Meg's. It can be a gig whatever.

G

Whatever for your place, you would want, but what this enables is now your applications can actually go over said limit, which in most cases might be fine and as a result, you would be able to create a kill switch as well. So if there's going to be applications that might go over, you know that's fine, if they're teetering along that line and depending on how you have your quota limits set up, you may not need to worry about worry about this at all.

C

Have one question so when I say talking about softly and in hard limit, no.

E

C

Allocation perspective should we use hard limit to do resource allocation and allocating at war using soft limits.

C

I guess how do you mean like, for example, I from meso standpoint view like you, have a bunch of discount agent and mixes just allocating those disk resources to every single framework when marathon, for example, has a one framework accept an offer for and he wants to use a disk and and the user? Actually, the user of America I just had a softer limit on the disk of rock on and I say one gig and I say the Harlem is like 1.5 gig.

C

So from our allocation perspective, do you actually take 1.5 gig and to reply to offers say: hey I only use, 1.5, gig or you say, I only need one gig.

G

Yeah, so it would, it would still be. The functionality would still be that you are reserving that higher limit, so that part wouldn't change. Okay, what this would enable is somewhat a little bit obscured from the user. Right is I want to use.

G

You know if I'm, if I'm asking for a hundred I'm, possibly getting more than I'm asking for, but that's because at the level of providing offers you want to like from a report from a resource perspective you, you obviously want to only allocate as much as you have available about, and but this this allows. This allows the slack on the on the applications that are running or the containers that are running.

C

Okay, so that okay, so if I understand just correctly like like say on that, the user is asking a hundred and I will ask Maysles for a hundred fifty and like the fifty being the Headroom, and you want you in forced Dakota, at 150 as a hard limit and and the soft. That means a hundreds. Like he's really waiting for soft. Let me exit XFS, yes,.

G

There is okay, so this is something that's built in functionality, um so I've actually started a little bit on a POC I'm, just trying to prove prove it out and I have I, have it to the level that I can set soft limits right now, I'm, just using a default. So in here there's the idea for flags that I have is you have a hard limit offset since I, just whatever they ask for you know, default is zero to provide. You know functionality as people expect right now.

G

So nothing changes in your face perspective, but if they want to set it, then they can set it above and then there's also the two other flags that are related to actually killing the the applications that don't want to or killing the applications that are over there their grace period. So in XFS there is actually an ability at the project level to set a a grace period. Default is seven days like that is the standard. So this would just be a false that you know.

G

Basically, all these are work as you do currently, and so once you enable this, and this what's.

D

Up, could you remind us again so forget about my sis for a second just a the behavior of XFS soft limit and place grace period and the heart limit? Okay,.

G

Yes, of course, so I'm XFS at a base level has three potential ways depending on at the latest. Version has three ways that you can do. You can do it in a user level. You can do it at the group level and you can do it at a project level for maize O's itself, we're using projects. So that's that's what applies here when you set a soft limit. It is something that then starts off a timer. That timer is gonna, be based on what what grace period you decided on for your application.

G

The grace period is a set as by default seven days, and what happens is after that seven days ends. Your sock limit becomes a hard limit. Okay,.

C

G

The grace period- essentially it will I mean you- can set it down to a minute, which is what I've been doing for for my testing purposes, but once that grace period is over, then it will block you and when you're violating that.

G

So, if you're, if you're, trying to query from a API perspective when you're querying this information, the way that it's represented is actually an epoch timestamp. So, regardless of.

G

So as as soon as the soft limits broken and XFS knows about it, when you query for that, that field is that grace period field is filled with the time that the grace period is now over.

G

So you know if it's right now, it's 655, you just now broken your limit, so it'll have it for 656, and once 626 occurs with this particular flag turned on, it will actually create a container limit and will kill the application at that point. Right.

D

So the goal for this changes so I'm wondering so what the behavior do you want? Okay,.

E

D

The when the tasks is between the supplement and the heart limit.

G

So when it's between the soft and the hard, essentially it so it depends it's time based right. So, if you're with, if you're within that grace period, nothing should happen. It should function exactly as as expected.

G

Your app can use all the way up to the hard limit of space, and you know, unfortunately, you know if your well actually, fortunately if your app is still growing, for whatever reason like beyond whatever you've. Even even beyond that, you know, it's not gonna, fill up your disk space, so you're safe in that aspect, but if you have, if you've then used up your grace period, if you and you haven't, met your soft limit, you're now blocked from using more space and as soon as the as soon as the watcher well I.

G

As soon as the the mechanism that checks sees that you have violated your since that you have violated the grace period, it will kill it with that container limit. Actually.

D

So the ultimate goal for use to kill it right, yeah and that's.

G

Not achievable.

D

When you only have the heart limit, correct.

G

Big well, it is, but then you get into this weirdness where somebody at because you could. You could technically have that hard limit be enabled right, but then you you can have the hard limit enabled. However, you can never break the hard limit, because it's a hard limit right right and so then you have to wait.

G

Sorry go ahead, then.

D

The behavior be that you go basically to the exact limit.

G

That's not always possible because the problem, the problem with how that is, is its debate. It's based on how much your application is trying to educate at time and if it's over, so if you're at just use simple numbers, we're at a hundred again and you are using 99, but you want to allocate even two.

G

Then it will just. It will just deny that allocation right off the bat, so you'll never actually get to that limit, because you're asking for two and I'm I only have one available, so you'll you'll never actually reach that limit and we do have kind of a hack.

G

We have a wrapper that basically happens whenever any application runs in marathon and it basically starts up its own thing and basically says if you're, if you're at like 99 percent, then then kill it, but we want to provide people the amount of space they're asking for and really, if they're going over that, then we know that you know hey they're they're, not doing something that they that they should and and we have certain applications that are that.

G

Are you know back and forth in terms of how much they use so maybe- and this this comes into the second half- is potentially setting up to have soft limits that are.

A

G

A

G

Soft limits that are customizable per container so that you could have add it like disk is disk, is what it is right and having some sort of soft this limit asked for, so that you know, even if you have this hard hard limit offset whatever you have here in this case, you know, if I asked for a hundred and I actually rep, you know ask for some soft disk under that, then I will get this functionality.

G

This table actually right.

D

I think so that customizability is because you would think so yeah. If we have that kind of limitation, then if you have a sets soft limit, but then your user application actually try to allocate beyond that range. So it's like no matter how much the gap is. You could always like try to allocate a wider gap right right.

G

And we want to have some sort of individual like we want to have a global rule for the purposes of well, most apps, don't really care they're, probably gonna use some amount, but this at least provides a mechanism for them, knowing why their applications not working I, was actually subject to this before so I'm working on a team temporarily to help with this specific issue, but I was I, had a started in marathon and I didn't even know what the problem was, because it couldn't get passed starting up with the application, like I couldn't start up the application, because the file that was being executed was bigger than you know.

G

What was bigger than the space that I had requested. Unknowingly, like I, didn't know how big my actual executable was at the time, and so I went to try to run it and it would just die and I had no explanation for why that's happening so this would this would help alleviate some of those sorts of issues as well, as you know, as well as others.

D

Yeah I think we're almost out of time, but for that.

G

D

Have a short comment, which is: if you're, the implementer of the framework, like you think well, I can customize it, but then, ultimately, that they are this different group then who's. Writing the end-user application right, so not always that the framework will be able to set the limit for that and the user application mm-hmm.

G

D

Empirically, it's just to me, I, don't know whether how wide of a class of problems in this can solve or really is just you have this one framework. That's specifically for this one application I think that might work.

C

Yeah anyway, I think this is kind of additive to the existing functionality and yeah I mean and those selfs in the use cases that you guys have I, don't see any problem implementing this. If it's not too complicated, I assume it's not too complicated. No.

G

No, the the POC was relatively simple, more or less. Currently, the XFS isolator does not use the watch. Yeah.

C

So that's why I figured that you probably need to implement down yeah.

G

So that's that's been implement or like it in my PFC I've implemented that and then that and then there's an additional function that is called asynchronously from the watch and then I have continuously yes.

C

So one suggestion I have is like I think, no matter what chance you did to make sure that the people can still use the old behavior like like, so that it can choose to use either to use the new behavior use the old behavior. So right, because area.

G

To container in.

C

G

Everything that I've been trying to add in here that would have default. You know so the offsets default to zero. So essentially, if you have a disk limit or if you have your hard and soft, if you don't have a hard and soft then- and you don't have this set to anything, but whatever the default is function at current as it does currently okay, so the watch won't be triggered yeah. The watch won't we, the watch won't be triggered okay trigger because of this.

C

One okay I see: okay, so I think who's gonna Shepherd that work. Have you talked to anyone that that my you go to Shepherd this work I think either yeah or change each yeah.

G

I started talking with James speech um so I if I can get yeah if I, if I can work with them, that that would that would be good yeah.

C

Okay, unfortunately, is not here, but I think you should reach out to him. I think he's the one that wrote the original XFS isolator so on reach out to him and ask him to review those patches will be great. Okay,.

G

What's the one question I do is in terms of submitting the patches? Is it through because I don't currently have a Garrett account, so what I just do I get email and send them to them that way or what it's log.

C

I think we do use apart. So there's a dock side about there's a talk about. How do we contribute caches to approaching maces so on the use.

E

A proxy review.

C

Board, but github also works if you prefer that okay.

G

Yeah either way is fine by me: we use Garrett a fair amount. So, okay, all.

C

Right: okay, that's good thanks! Thanks all right, I think we are over time, that's um wrap up here and see you guys in two weeks. Thanks thanks, guys see you guys yep right.