Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Local Ephemeral Storage Resource Management - Jing Xu, Google

Description

Local Ephemeral Storage Resource Management - Jing Xu, Google

Currently Kubernetes does not support storage resource usage guarantee and isolation like compute resources such as CPU and memory. This talk will present out effort for improving Storage Resource Management in Kubernetes with focus on capacity isolation in ephemeral storage. It will explain how we support resource guarantee and isolation at node, pod, and container levels.

About Jing Xu
Jing Xu obtained her Ph.D. from Electrical and Computer Engineering Department, University of Florida in May 2011. After graduation, she had been a lecture in School of Computer Science in Florida International University for about 4 years. She moved to Bay area in late 2014 and joined VMware, working on Storage Policy Based Management project. Later in 2016, she joined Kubernetes team in Google and now is working on storage management.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hello, I think we can start. Thank you for attending this talk. My name is sting I'm Google engineer. Working in community is storage. Lifecycle team today, I'll talk about resource management in kubernetes, focusing on local ephemeral storage.

A

Strange thing is the the words that the bush ad hope. You don't bother you as much, so this is the agenda, so first we want to make it clear right why it's important for locally former storage resource management. What problems we are trying to solve and I will give quick overview of resource management in kubernetes and the general resource model supported in kinase and then focused on the storage and how we manage the locally for storage at different layers and how we solve the different problems at last. Give a few directions of our future work.

A

I know today's the last day and almost lunchtime I'll try to make it on time. So in case you have more questions and want further discussions. These are my contacts, so my kid hub idea is a little bit different.

A

So first, as we know, no communities is a solution or ox trading containerized. So were closed. So when you have application you're closed, a scheduler will pick a node to run the containers for applications over time right as a user. You might experience something not very present like suddenly. Your container just are killed, terminated or you notice your service running in a container just getting slower and slower or even worse, the whole machine just keep rushing and all your workloads running they're, just in trouble. So why?

A

What's going on keep in mind that resources in cloud environment are shared? Containers are running together, side by side in the same house and they compete for the resources.

A

What if our container had suddenly use a lot of CPU and memory and cause other containers that run very slow and also even get killed, because out of memory or a container just produce a lots of data in general lots of files and cause out of disk error. So we need to solve these problems with proper resource management. So, typically, we have different resource. Manage goes to address different aspects.

A

First, the efficient allocation of resource, so we want to allocate just enough resources for applications and so that you can have some performance guaranteed and also we don't want to like I, don't care too much with our money and same time. We want to avoid like over commit resources, because it will cost failures, downtime and hurt your performance.

A

The second aspect, the second goal is since containers are competing for resources and we want to have some isolation among them so that we don't want to have some workloads use up all the resources, and the third aspect is for kubernetes right. We have some critical system processes that cool it is running as a daemon on each node and it managed or were closed and those critical system processes also need resources and to make sure systems the parity we won't have. Those critical processes also have enough resources.

A

So, in his resource, that definition is something that can be requested can be allocated and can be consumed and typical example of such resources. Cpu memory, we call CPU compressible, because a CPU can be started as needed, but memory cannot and in plenteous we support a simple resource model, so allow user to spice by resource request and limit so request speedily the amount of resources you want to allocate for your application file container and how much you need and the limit set upper bound that how much resources you want.

A

You can be consumed by application and we call those specifications as they they are state because it's described how system should behave right, how much resources you want to allocate and consumed and the actual resource usage we called extra state and in Cabrini's we have control up to drive the actual state always towards to the desire states as close as possible, and that figure shows for second request an image we allow different values.

A

The reason for that is because, in many cases some workloads are first of all and so most of time they only consume certain amount of resources, but a kidney. They might have big spike. So we want to allocate the resources that these application take reuse so to have better utilization. We are resource, but we also want allow it to like conversed, occasionally and the actual usage might be different right.

A

It can be below the request or can be more than request, but it should be below the limit, always with this simple resource model like let's see how we allocate the resource in a solid resource. So first each part will said request and the scheduler will check all the knows, not the available capacity, whether it can satisfy the request, and if there are multiple candidate notes, can all satisfy the request. It will rank or the nose based on the scheduler policy and pick the best one.

A

So we support it for in scheduler policy, some like more to us even in distribute or close, some more like what was healthy pack. They were closed and by allocating enough resources, as you request, we kind of reduce the chance of our committee resources, but it might still happen we'll see a little in later slides and also you can set resource limit to make sure the actual usage under limit. So the system will keep monitoring the actual usage and take actions if it is see any violations, so for CPU it will start all the CPU.

A

So the CPU users should never go above the limit and for memory it will kill the container, so the memory will be freed up and in a pot, as we know, when container is cured, the power will automatically start a new container. So continue the work.

A

So so far we talked about the resource management for CPU memory, how about storage and in the following slides, we'll talk in more detail about the storage management, so for storage company support, many different type of storage in container that you can write, read data and those data are are temporary. So when the King dies finish so the data will be gone and those data are typically stored in the local disk and no problem is the data in one container cannot be shared to other containers. Since part consists of number of containers working together.

A

We need a way to share those Dana, so we have a volume concept in communities, so a part can have different volumes and each volume mounted to different containers and continuer can read right to the volume and share data, so one type of volume we called am reader, so this tab volume is also considered as ephemeral, because when the pod finish that terminated the data in that empty row, volume will also be clean up. So it has the same life cycle as their pod.

A

Typically, empty volume are used for store some temporary data for caching and Postgres page or you want to check pointing or long like running computation and this data when Polly finished that is okay to clean them up. There are a few other types of volumes like secrets, configure map. They are basically wrapped around to empty their volume, so they have. They are opening to the ephemeral volume.

A

But if are, they will be back up to my local storage too, and we also a support persist ammonium, because people need to persist their data so for percent volume. Typically, they are backed by dedicated disks and it could either be remote like gcpd, either live it address EPS or recently. We also support dedicated local disk, and there will be a talk in afternoon Jericho by Michael, dicker and Michelle. Talking about support a total precision disk and for such persistent volume it will have a life cycle totally different from their pause.

A

When the pod finish that the volume is still there that they have will be persisted, and normally we use called persistent modem. Api object to represent it if a PHA need to use that persist, data it need to refer claimed, recalled PVC and PVC Pines to one of the PV, and the part can use that precision model service rating we can see a former storage red are shared between containers among containers and empty the volumes. So these results are shared. We also need some management for their shared resource and for persistent imodium.

A

Typically, they are dedicated discs. So we don't need to worry about too much so for locally for more storage measurements. We also want to achieve these different management goals and I will talk about in more detail how we achieve it at different layers.

A

So just like CPU and the memory first, we want to support a former storage also as first-class resource, so that user can specify in the container specification how much you want to request how much you want to limit for a former storage usage, and here we have just as example. This part has two containers: each one requests different amount of resource and different limits and for request.

A

The scheduler will check whether the node has enough capacity to schedule the the container in the pot and for limit its monitoring, the actual usage of the container, and if they see it, exceed the limit, it will evict part by eviction. We mean your terminus departs, gracefully so that the emitter bottom, like data or the continual data will be opening up and the disk storage will be freed.

A

Now, considering as a part right, we just mentioned, the part can have like volumes and besides, we want to constrain how much resources used by container. We also want to constrain how much resources should be used by the volumes. So we add a field the size limit to visit a gave upper bound like how much resources can be consumed, that is volume. And similarly, if the monitoring like detect the bottom usage exceed so it will be picked apart, and by setting the limits for container and also volume read.

A

We have good isolation among different containers and also the past.

A

So one thing is right: now we don't support explicitly pod level like resource specification, resource requirements, but internally, like in system, actually calculate the power level resource. It is basically the sum of all other containers, so here in this example, we have two containers.

A

So at the pub level the request would be the total, like twelve gigabytes and the limit is 14 gigabytes and the usage for this part will be the total usage for come from auto containers and also volumes and meter volumes, and we want to make sure at pod level the limit should be enforced.

A

So the reason we also care about power level resources pad level resource consumption is not simple ages, aggregation container strength. It also has some pod level overhead in the future we might want to support. Like is pasilla pod level, specification resource specification.

A

And by citing request limit values we can also cast by the path to different cue as classes and the first one. If you set your request as the same as your limit, we call them guaranteed pass, so those paths right guarantee have this much resource and also they should not use more than they request. So those part will not be cute in case of like resource contention only if the usage exceed their limit and also as we mentioned, we want to support like a portable workload.

A

So when you request is smaller than the limit or at limit is higher, so we call them first of all workload parts they can use more resources, then they request, but they are more likely to be cute compared to guaranteed and the last one, if you don't specify anything for your part, we call best effort, so those part can fit anywhere so easily. They can use any available resources. But if you need to even pass to reduce the contention, then the best effort part will be the first target.

A

So now we covered a container level pod level, resource management and are considering know the level rent, although we have like limit for container and past, but we might still have issue for know the level resource management, so here use that, as example, first based on a capacity, we schedule Sri pause and one is guaranteed ones, worst of all ones, best effort, so based on the requests that they can perfectly fit into that node, however, and worst of all and best effort, part can use more resources than then they request.

A

When that happened, we can see definite. The capacity is not sufficient anymore and we he'll have like out of disk problem to solve this. We have an eviction manager to detect whether there is this pressure and take actions, and also it allows you to sell some like eviction threshold, so it monitors okay at node. If the available resource available disk space is smaller than some amount, like one gigabyte, it will start eviction action before in bigger pot. It will try to reclaim disk resource by TB kingdom and use images or delete dead parts.

A

If it's still not enough, then we will try to choose some part to evict. The order of choosing will be in the order of the queue, as we just mentioned, and here for eviction start hold, it can be solved or hard-hard means you take eviction action immediately when there's a violation. Soft means you allow, like a certain pure time for Chris pyridine.

A

So with the immigrants our code, which seems to like, have some protection for node level resource. However, we still have issues so can steering here example, and we have this capacity and we allocate we scheduled three paths.

A

All of them are guaranteed paths right, so we those paths should not use more than they request so right now, it's perfect fit, but remember we mentioned there are critical system processes running to and they also need resources, so they will like compete, the same resources like disk space with all the other parts, and when this happened right, we also have this resource contention issue and to solve this we have a concept of allocatable, and so basically it allows system Amin to reserve certain amount of resources for these critical system processes and after you reserved that much so we have a local resources speak today, the capacity those reserved and for users part they can only use the allocatable part, and you may wonder, like how much I should reserve for my part- and we need to make monetary in the note and checked the total usage and note level and managed the total usage of users part.

A

Then this would be the system overhead and we can roughly estimate that over here should be proportional to the capacity, because the bigger capacity you not have you expect, have more closed and the system overhead is roughly like proportional to the were close. Let's say: cubed, for example, kübra usage will be roughly proportional to the number of parts you're scheduled to that node.

A

So, after that we have this allocatable results, you can see the Sur part p3 no longer fit, and we can only schedule 2 parts based on the allocatable resource.

A

And we see right for scheduling part, we make sure we have a schedule pass based on the eligible, and we make sure the system demons will have enough reserve resources. However, again that scheduler part is based on the request and part can always use more than they request. When that happened right, we don't have enough like we cannot guarantee the system. A process have enough resources, so after para scheduled running, we also keep monitoring how much each users part are consumed.

A

If the total usage will exceed there at a capable part, the eviction manager will also take actions to evict parts.

A

And we mention the rats, the eviction pics ranked their part based on the Q s, but Q s. Catification is only based on the setting of request and limit and whether the request is bigger than limit or not. It's not very flexible. How about you have some first of all were close that very important.

A

You don't want to be victim first, so since released white community is support, alpha feature or parts pure ret, so the priority will indicate how important of your path compared to others and the eviction policy right now will incorporate the part priority. So first, it will target on a part that the usage is more than they request and they will rank their paws based on the priority.

A

If there is there is a time, then it further will run cause based on the difference between their usage and the request, and here we show example how you specify the priority for pot. So you have a priority class if you object- and you can have different priority here- is in this example, we have high priority and in the pots back, you need to give. What is your priority class name to indicate how important your pot is?

A

A

What happened to the containers into the pot, so the Kenya we are all so cute, but if your part is managed by, like some other controller like suitable size or deployment replica set and those controller will start a new part at a different place to make sure you have passed always running, but if your part is not managed by any controller that and after eviction, your part base today will be terminated and there's no work, yeah again the data.

A

So if you use persistent volume like I said so, even part is terminated, the data will be still in the volume and theta will not be clean up, but only the cathedral volume that is recalled a former or like modem and the data will be coming up and if you have other parts it will, it will can continue to use the new part when they started. It can continue, use those per system, all you. So the data will be still there. You can still access those data.

B

A

So yeah for persistence. Basically we are saying you use some dedicated disk right, it's not shared, so basically you can kind of make sure your testicle size is enough for and also we plan to have. We are currently working on some feature like a resize possess the volume, so it can be done like resized this volume on time then dynamically. Yes, there's a new feature that we plan to support.

A

A

It's kind of nuts, really it's kind of mixed. First, it will check whether the usage above request right. So in other case, we are more targeting on portable and best-effort were closed for guaranteed right issue, not the actualization, not above the usage.

C

A

Yes, current policy is yes, it's kind of debatable yes, but the behavior is like that. Yeah.

A

Okay, we covered like resource management at continued level, Kitchener pod level, node level and we seem he'll, provide some allocation and isolation bar resources. But for like in cloud and often we have different group of people. Different teams share resources and how we can make petitioner allocate resources among different teams and in CUDA is we have namespace and namespace pezzi. They allow you to partition resources among different group of people and so for namespace. It allowed to specify resource requirements for each group.

A

There is a just example, like you create three different namespaces and when the part is created you specify which namespace this part belongs to and if you don't have any like home, now create any namespace by default. The part will assign to a default namespace.

A

And how to like specify resource requirement in in space, we have a code objects. So in connecting you, you probably know des right. Everything is a pair projects, and so, if you create a resource code on the object in one specific namespace, you can in a spec, specify ok how much it is required, how much limit and that it is specified like how much parts in this namespace can request.

A

And how much is the limit you want to set for all the paths in this namespace and when part is created in this namespace and the system will check whether it violates the code huh assigned for this namespace.

A

So by I have this restriction right. We basically kind of partition the resources among different namespaces and also, if you have a resource code on in a namespace, then all the parts in that part required to have resource limit and requests specified so to help user to have some default value. We have another API project called limit range and you can in this limit range.

A

You specify some default value for a former storage here, as example, and when the part is created in this namespace and if you don't specify anything ready for they will apply to those default values.

A

Okay, in summary, we talked about like how we support locally for storage as for scarce resource, and we talked about how it support the management at container part level. So you can have like resource allocation and limitation. Note level read by a loop by a low category concept. We can ensure the system, stability and a namespace level right have a partition. The resources among different groups.

A

And for future work first, so far we more talked about the disk space allocation, and this guy open way is also a resource shared by other containers, and so it's really very nice to be able to pass it. Yes to allocate resources or have some kind of resource management for disk I/o. It's very challenging. We know and we are still discussing and whether we can support it and how we can support it.

A

And now, since we add this locally for more storage, resource management will extend our magic API so to allow user to like very easy to more how resources are allocated and how resources are consumed and, as I mentioned earlier, we want to also support the pot level limit limit and request a setting, because it's more convenient for users just consider as potable and also it allowed the containers instead of how to share resources instead of like a fire agree of isolation among each individual container and so far we talked about resource management way.

A

Also, we only set the requirement aesthetically right before pot great, you set a value and when the pot is running, you cannot change that value, but in many cases the workload stretch just keep changing over time. The value set may not appropriate anymore. Then you will lose the benefit of those resource management and we should consider at how we can dynamic manage resources. So we allow you that I may change those request, citing and based on the application, behavior and also for system processes range if they consume resources, and they may also change over time.

A

Rather bit smaller clothes come in the system, overhead will become bigger. The the initial setting may not enough good enough, so we want to like, based on the system behavior. How to dynamic is that the reserve resources and a low category sauce.

A

And this work is a team work. I want to acknowledge our communities, hew members and also community contributors. So when it is open right, if anyone inches, you can also contribute. Okay, that's about my talk. Any questions. Yes,.

D

A

Yes, so right now, bananas itself does not provide any analytical tool for this purpose, but I think during some talks. I noticed there are some people like working on some this project to analyze the resource usage and to give you better estimation of your application and also we plan to support some dynamic sighting and in this way use that someone Erika all right. You can better estimates the resource usage, so renauldi hakuna itself does not support anything. Yet, yes, okay,.

A

A

So right now we only focusing on the root filesystem right and pay. Today we only monitor how much available in for the root filesystem and we don't consider other disks. If you do have that, and you can kind of have to manage yourself to make sure you have enough capacity yeah it because in in the typical setting right, the local former storage is backed up by the root filesystem yeah.

A

A

So busily like say, we have example like specify employer right in the path spec, you directly specify that volume, and by default those volumes are backed by their local disk and persistent volumes. Those are typically remote disks like enclosure gcpd it as EBS, and we currently also start supporting local disks, but there should be in a separate, dedicated disk and this how a like managed, persistent volumes and they separate from theirs locally. Former storage yeah.

A

A

Like I can't say, I began the.

E

A

E

A

Provisioning no right now we don't have any work done for the thing. Provisioning, I think yeah.

F

Unfortunately, or like static posting or like dynamic, resizing like dynamic, static conditioning, for example, the plan is too late. I think the plan is flee, keep it completely separate. So we probably see people like building operators for different sets of constraints, because there's no one storage solution that people like so it's gonna like keep it separate.

A

I see someone in the back have questions. Oh yeah, any other questions.

A

A

G

The question was, instead of using, do you to measure the storage usage of a pod or container? Have you ever considered using projects quotas in XFS or extension for.

H

G

In XFS, I thought was intrigued: yeah before I extension for my dilatory, but for.

F

Exercise I think yeah.

F

Short answer to that is like, yes, we have been thinking about it for two years, I took a while for ext4 support tool and it wasn't for ten I think and even then, like the the user, space tooling isn't complete. Yet, like all the cooler tools, it's about the plan is to have like part level like you, basically need multiple project IDs for a given pod. So we want to get that.

F

But there's like you wanna enable people who are like have them do what they can do with the level of tooling and access today before trying to go to ask Roger code up there, but just like, if you want to get that get up to you, know.

F

We don't want to go the ldm road because it's like just totally all the way to the bottom line, stay as close to.

H

F

If he I agree, you doing that like if he had a separate file system, that it would be great.

D

But there's also like scoffs.

F

So there's no single storage.

H

You know, can you make ephemeral storage? You know driver plugging pluggable, so I can use LVM as someone may implement project waters.

F

My short answer to that is pilot issue.

G

F

But then you don't get you don't get the you know.

G

F

All the addictions and, like you, don't get resource tracking and liberation.

G

That way you get this under.

F

G

E

You know separated at post level. Yes,.

A

E

Evaporates at the first level, it's a rule.

E

F

The fundamental philosophy so.

E

Let you not leading this to explore.

F

The design philosophy is like over on your network for, in general, like so, you have storage separate to compute and you use crash only when you absolutely need it, and most of your apps might not need it. They use it for caching or like logging,.

A

Yeah, that's that's the problem we want to address. Yes,.

D

Thank you. Thank you.