Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes On Cgroup v2 - Giuseppe Scrivano, Red Hat

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes On Cgroup v2 - Giuseppe Scrivano, Red Hat

This talk will go over the current status of cgroups v2 in the Kubernetes and container ecosystem. Efforts are underway to enable cgroups v2 in containers runtimes and up the stack in Kubernetes so users can benefit from new kernel features such as PSI and have better OOM handling through using projects such as oomd. Particular focus will be placed on the changes required in the OCI (Open Container Initiative) specifications and how the containers runtime must be adapted to use the new version.

https://sched.co/ZeoS

A

Hi everyone I'm joseph esquivel, I'm working reddit in the containers team. During this talk, I will go through the secret linux. Kernel feature, give some basic introduction, with a particular focus on how it's used by kubernetes I'll go through about c group version and what it took to get support for c group of v2.

A

So what are c groups c group is a kernel feature that allows to set research constraints to a group of processes. It also allows to monitor how much of these resources are really used. Its structure has a set of different controllers, each of them specialized for a kind of resource. So we have controllers like cpu to limit the monitor, cpu usage or the memory controller that is specialized with memory usage c groups are structured in a hierarchical way. Each c group has a parent, except for the root c group.

A

They are managed from user space to a file system. So how do we use them? Whenever you have a problem like limiting resource usage for a pod or a container c groups can help us. We can solve problems like having constraints on how much memory can be used, how much cpu and what cpu cores are allowed for some containers.

A

We can set also weights on each c group saying how much time in proportion to the other c groups it should get when there is cpu contention. We can also limit assets to the os devices. Also pits are a finite resource, so through c groups we can also limit how many of them are allowed to each container.

A

uh To give some background, sigrpu v1, the one currently used by kubernetes, was first developed at google in 2006 and uh force finally merged in the upstream linux. Kernel in 2008 secret v2 followed up a decade later and was released in 2016.. So it's uh it's not really such a novel technology. It has been around already for some years.

A

The proposal for adding uh secret v2 supports was accepted just a few months ago last february, and the secret pv2 support was added for the release. 119.

A

c groups are not a monolithic feature controllers were that, the later after the initial support, for example, ldma was added to c group week, 1, even after zero pv2 was already released.

A

So let's start with segreg one, as I have already stressed it, it's a group of different controllers, each of them specialized for a single resource configuration separately for each of controller.

A

They have a hierarchical structure. In the picture we see a few c groups created for the memory controller. A and b are at the same level, while c is a child of a so it inherits all the limitations we we set for a and the likewise d inherits all the limitations from from c child c groups can restrict further the limits that were set for the parent.

A

Other controllers might or may not have the same urge since they are configured separately. They could have a completely different uh tree in practic. In practice, this freedom is never used and and the same hierarchy with the same naming is mirrored across all the controllers.

A

Later we will see how this experience led some decisions in the c group of v2 design, since the freedom of having controllers configured separately bring some difficulties in managing them. While a container is being configured, it's part of some c groups, but not of the others. So setting c groups for a process is not an atomic operation.

A

So cdc group there are, there are no syscalls or io ctrls to configure c groups. Everything is managed through the file system.

A

The c group file system is the api for managing c groups. Each controller in secret v1 is a different file system. It must be mounted separately well. This is not something users must do manually. Usually the operating system takes care of setting up all the all these mounts, and we we see how how many mounts there are typically for a on a c group of one host, like each controller, has a different mount.

A

Some useful information is also exposed to the procfs file system. If we want to know what uh c group process belongs, we can look that through the uh proc b2c group file, there is also another file that gives more general stats about c groups usage so how to interact with c groups.

A

Regular file system operations like mkdir rename write rmd, are used to manage them to give an id in this table. We can see what operations are allowed to use.

A

Let's say we are running a process and we would like to limit its memory usage, and we want to do this manually using directly the c group api. So the first thing we we have to do is to make sure the memory controller is mounted.

A

So under the memory controller, we create a new c group uh for simplicity, we'll just create it under the the root.

A

Then we will move the process we want to limit under the secret we just created, and we do that by writing the process speed to the cgroup.prox file and finally, we can set the limits on the new c group by writing to the memory.limitingbytes file.

A

Once the is terminated. The signal from must be cleaned manually through rmd. So no one is going to do this manually, but these are exactly the steps the container runtime takes for managing c groups. I need to quickly introduce quality of service classes in kubernetes, as it is important to understand the next slides. The cubelet assigns a qs class to each bot, depending on how its limits and requests are set.

A

If each container in the pod has bought limits and requests set and they are equal, then it gets what it asks for. So it gets assigned the guaranteed qs class if it's not guaranteed, but there is at least one limit set deposits assigned the boostable qs class, everything else that is neither guaranteed nor boostable. It's a best effort.

A

So the human killer, strictly speaking, it's not a c group feature, but it's used by kubernetes to complement how it's used c groups. Actually each linux process has a home score in out of memory situations.

A

The kernel will take it into account and will prefer terminating processes with a higher room score, it's possible to modify its volume part to a configurable adjustment.

A

uh So uh um scoring kubernetes, each qs class is assigned a different home score, as we saw earlier. The higher the score is the most likely the processors are going to be terminated by the kernel. If there is not enough memory best effort, pods are the first ones to be terminated.

A

Guaranteed pods instead have almost the same home score as cubelet. The boostable pods instead gets assigned a dynamic value that it's calculated, considering how much memories are located to them in proportion to the entire available memory.

A

So c groups are used by the cubelet together with home, score, to manage and organize the pods. The hierarchy created by kubernetes is quite complex. There is a cubot c group, the dot slice suffix is used when using the systemd driver.

A

All pods are under this three group, it's possible to settle limits for all the pods. At this level.

A

There is a mechanism in the cubelet called the system reservation where some resources are allocated for the rest of the system. When such reservation is in place, limits for all the pods are configured at this level at the cubot c-group, so that it will affect every every bot.

A

The guaranteed pods are a children of the cubs c group at the same level like uh as each guaranteed pod. There are two sibling c groups, cubs best effort and keyboards burstable. Each of them contain all the pods for the for the qos class. They refer to.

A

The the the difference at the c group level between uh burstable and best effort is that best effort gets the lowest cpu weight so that on cpu contention, the best effort.

A

Pods gets the lowest amount of cpu.

A

When we specify a limit in the yam spec file, they will be ultimately written to the c group file system, as we saw earlier until it gets there, though, there are quite a few steps involved once the pod is scheduled to unload the cube blood handles the request of creating the pods.

A

The limits that are configured for the pod are passed down to the cri runtime using grpc and, together with a home score value that the cubelet calculated.

A

The container runtime after it sets up the storage and the network for the container it passed down the configuration to the oc. I run time that it responsible for for running and configuring, the container c group.

A

A

We have the communication between cubelet and cri runtime using gerpis.

A

Then we have the container runtime passing the configuration to the ocran time through a json file.

A

The ocean run time in the in the steps to create the container will uh will write this values to the to the c group file system and set also the room score adjustment to the proc file system.

A

The cubelet uses c groups also for monitoring resource usages and that's done through the advisor. The advisor is a package embedded directly into the cubelet.

A

So while the container runtime is responsible for creating the c groups for the various containers, c advisor uses the same c groups to read: statistics on what resources are used by each of them. I won't go into the details, but obviously c advisor also needs to understand the c groups version used by the host shortly- it's just another component to take care of for the c group v to support.

A

So why do we care about c group? We do if we can do everything already with a single moving one. Well, there are some issues with the sigrupui one that ultimately led to having a new version of for the for c groups, as we saw, each controller must be handled separately, even if this freedom is never used in practice. It is just another uh additional complexity for uh no real gain.

A

Also, integration with some subsystems is not ideal and fixing. uh Some of the ps is not possible, as it will be a breaking change.

A

One big issue that doesn't affect directly our kubernetes groups today is that delegation is not safe. It's not possible, with c group v1 to delegate a subtree to a less privileged process in a safe way.

A

It's also not possible to pre-allocate resources if we want to pre-allocate for a c group a certain amount of resources like we like. If we want to allocate uh some memory, that's not possible today, with zero v1.

A

As we saw, the setup is not an atomic operation, since a process must be configured for each controller separately.

A

uh Another big issue that it's affecting kubernetes is that the um killer is not c group aware.

A

If there is a out of memory situation, the and given different process with the same um score, the kernel will terminate these processes without taking into account to what c group they belong. So what happens is that uh processes from different containers can be terminated so leaving the these containers in a broken state, since some process are terminated, but some others are left running.

A

Ideally, what we would uh want is that a container so that it's under a single c group is terminated like a an atomic unit, so that all the processes in the c group are terminated and, and also we would like to first try terminating a single container before uh affecting uh more of them.

A

So, let's get to singapore v2 now.

A

So how we got here uh so fedora 31 was the first distro to enable secret v2 by default. We took a bet with it, but it turned out to be a good decision.

A

This was the turning point since once we switched to the default for fedora, the rest of the ecosystem followed, the cost was uh was high since.

A

Since programs missing support for secret, we do wouldn't work on a default. Fedora, 31 installation.

A

Cyrano initially was my side project. I started working on it as a replacement for run c, while playing with it. I added support for c group of v2 and and it gained features until we decided to use it by default on fedora with podman.

A

We got a good feed the feedbacks with that, and it's still used as the default oci runtime at the time of the fedora 31 switch kubernetes also gained support for huge tlb.

A

That controller will still missing in the in the kernel for c group v2. So by the time I got to work on the enhancement proposal for the kubernetes. I also needed to make sure that huge dlp was supported in secret, v2 soi, to to add support for uh tlb in the kernel.

A

So why do we need secret v2?

A

Well, there are various reasons, as we saw before, with the issues related to singapore v1.

A

The most important uh one for uh having the the switch to secret v2 is that secret one is considered legacy and, as such is named in the kernel. Also, no new features will be added in the future, so this uh this migration had to happen at some point.

A

uh Also, all the issues we saw before with the c group v1 are somehow fixed with the group of v2. We have a c group aware whom killer so that containers can be terminated as.

A

Atomically and my favorite feature that we gained with c group of v2 is that delegation to less privileged process. It's safe. What does what does it mean? It means that uh we can create a sub c group and well literally shown it to a different user, and let the user fully manage it.

A

So that an user that it's not root on the system can fully manage a c group and that's and that's safe, a different return c group. We want another difference.

A

uh Moving to c group v2 is that on companies by default we also enable the c group namespace. I will talk more about it later so c group v2 there is a single hierarchy different than v1, and there is also a single mount point and the process is part of a single c group.

A

Some controllers are not entered by c group itself anymore, like the devices controller. Now that it's and it's entered through eppf.

A

The controllers are appropriate property of the c group itself. So now we don't create a c group under a controller, but we enable a controller for the c group.

A

A controller can be used in a c group only fits already enabled in the parent c group. This is done through a new file. Cgroup subtree control.

A

Not everything that was exposed by sigrpup v1 is available with c group v2. For instance, there are some stat files like usage per cpu, but it's not present in c group v2.

A

There are also some new features that are not available. In c group v1, some configuration changed semantics in v2, for example, the memory swap limit is configured in a different way.

A

Some other configuration kept the same semantics as in v1, but are using either different names or different ranges.

A

An example is cpu shares that changed both the name and the range in c group v2, um so other uh difference with single proof. V2, the first one. It's a huge difference to always keep in mind with c group. V2 processor can be added only to leaf nodes in the hierarchy.

A

Each node in the c group hierarchy can either have children c groups or f processes with c group v1. Instead process could be moved at any level in the tree, so we we had a c group that was having a process to manage, but also having sub c groups.

A

As I already said before, a controller must be enabled in all the parent c groups before it can be used in a in a c group. Originally all the trades had to be. In the same c group, the this limitation was relaxed at the newer kernels and now it's possible to migrate single threats to different c groups.

A

There is also an another cool feature, ns delegate that was added later, and that makes the allegation to privileged process even safer, as it enables additional restrictions.

A

Pressure style information is a new c group v2 feature for each c group. It says how much time was wasted waiting for resources to be available. It gives a good indication when the system is not doing enough useful work and if a pot must be terminated.

A

This new feature enables umd, then killer in user space that it's developed by facebook and being added to systemd, so that the hume killer management is moved to user space. From the from the kernel.

A

In secret v2 memory limits are configured through four different files that gives a lot of control on on memory.

A

Still, this configuration is not exposed to kubernetes as we as we are exposing at the moment, just the features that were also available in c group v1.

A

But this this is something to well to improve in in the future.

A

Now, with c group v2, it's possible to specify a portion of memory, that's never going to be reclaimed, that's configured to the memory.min file and there are also other levels: configurable memory low. It's a it's a memory that it's reclaimed only if there is nothing uh reclaimable in other c groups, then there is memory high.

A

And the kernel on the wrong run tries to keep a memory usage for the c group below this limit, and then there is uh well, there is still a heart limit, and once that is reached, the process is it's terminated immediately by the kernel.

A

This is not a secret feature added by secret v2, but this was not used the before with cgroup v1, as only well only newer kernels had support for it with c group v2. Instead, we are sure that the kernel it's new enough to have also the c group namespace, so it makes sense to enable it by default.

A

Also, this is the right timing for such breaking changes, because, anyway, we are changing all the c group api is exposed to the user space and to the containers.

A

When the container runs in a new c group namespace, it won't be able to read its full c group from the host perspective anymore.

A

It will be limited to the namespace that we are just creating when a space is created. The current c group becomes the the root.

A

The ocean time is the component responsible for ultimately setting up the container. It's the lowest level in the stack and, as we saw earlier, it gets the configuration directly from a container's runtime engine such as cryo container d, as a json file.

A

As part of the configuration that it gets, there is a resources block that specifies the c group limits to use on a c group v1 system, as we can see on the slide, this configuration maps exactly to the c group file system. There isn't any creative step in the oc iron time. It just moves the configuration from the json file to the file system.

A

There are time, specs are designed for c group v1. They don't map nicely to c group v2 different file names in different ranges, and some times the different semantics are not managed through this configuration, the only mechanism for extending the ucrm time configuration is true annotations, but that's not really nice for structured data and it's not a standard.

A

It's not. It's meant only for custom extensions, not for something like sigrupue to support the first two problems are somehow solved today and I'll show you how, in the next slide for the tiered one, I opened a proposal for extending the runtime specifications and adding native support for c group v2.

A

So the problems I introduced before are solved today with a mechanism of conversion between the two versions.

A

Firstly, I had implemented it for siran and they thought it was a hack but well it was good enough to get started without worrying about changing specifications and the rest of the stack.

A

An example is the cpu shares file that if we get the configuration from c group week, one that so the range as the c group v1 valid range is converted to cgroup v2.

A

So it's written to a different file and using a different range.

A

Advantages of such mechanism well, is that a container engine requires minimal changes, as it can generate exactly the same configuration for the oci runtime same thing for the cubelet that can pass down the the same limits as before, as they will be converted later anyway, uh run c now performs the the same conversions. So there are.

A

There are two oc iron times uh using the this mechanism. Now.

A

So what's next well, we reached the features parrot with cgroup v1. Now, uh probably the first next step is to get the runtime specs changes and once we have that we can add native support through the stack starting from the starting from the oca runtime and and extending containers, runtime and kubernetes, to take advantage of the new features we have with c group v2.

A

C group v2 enables some useful new use cases for kubernetes.

A

As I was saying before, delegation to less privileged processes. It's it's safe now. So what is possible for kubernetes that it? It will be possible to have nested the kubernetes with a full c group support and that's, and that can be done in a safe way, since the nested environment cannot affect the outside limits.

A

That's all thanks.