Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: What's New in CRI-O? - Mrunal Patel, Urvashi Mohnani, Sascha Grunert & Peter Hunt, Red Hat

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

What's New in CRI-O? - Mrunal Patel, Urvashi Mohnani, Sascha Grunert & Peter Hunt, Red Hat

CRI-O is a lightweight container runtime written exclusively for Kubernetes. In addition to being a standard component for deploying secure and stable Kubernetes clusters, CRI-O has the unique advantage of being able to tailor its behavior to the needs of the Kubernetes ecosystem. In this talk, Mrunal Patel, Urvashi Mohnani, Sascha Grunert and Peter Hunt, the maintainers of CRI-O, will provide an update about the latest feature developments, as well as live demonstrating typical real world use cases around them. In addition to a review of the basics of setting up and using CRI-O with Kubernetes, the talk will cover improvements around the handling of CNI resources, the ability to tailor container resources with workload types, and updates to the collection and broadcasting of stats and metrics. Join the CRI-O maintainers to learn more about how CRI-O works in action and why it’s the perfect choice for your Kubernetes cluster!

A

Hi everyone, I'm renault patel, I'm a cryo maintainer and I work for red hat. I also maintain run c the oci runtime spec and participate in kubernetes signal.

B

Hello: everyone, my name is irvi shimonani and I'm a senior software engineer at red hat as well as a cryo maintainer.

C

Hey folks, I'm sasha one of the maintainers of cryo and it's a pleasure to be here today.

D

Hey everyone. My name is peter hunt, I'm a software engineer, working at red hat, primarily working on cryo and other container related technologies.

B

So today we're going to talk about what is new and cryo. Since the last time we met at coupon eu first we'll go over the steps of setting up kubernetes with cryo. Then we will talk about the improvements we have in place for cni handling, as well as the different types of workloads.

B

And finally, we will talk about the enhancements made to metrics. So let's look at some of the basics of running kubernetes with cryo. First thing we need to do is install cryo on the host, we're applying to run our kubernetes cluster. We have package versions of cryo available from multiple operating systems.

B

As you can see here we have fedora open, susie, santos, 8, debian, raspbian and ubuntu. Packaging for cryo is largely driven by requests. So if there is a version or operating system that is missing, please feel free to open an issue for it.

B

You can also build your own binaries from source if you would prefer to do that. One example would be to get the latest and greatest cryo that hasn't been packaged yet, but is available in upstream.

B

We also have support for building static, binaries and our binaries can be found in our google cloud storage bucket, so once you've installed cryo on your system, either through the package versions or by building your own binary. The next step is to enable and start cry on the host so that we can run kubernetes with it.

B

Cryo versions walk in lock, step with kubernetes versions. So what this means is that cryo118 is guaranteed to work with kubernetes 118 same for 119 and so forth, so make sure the versions of kubernetes and cryo you're running match once this is all done. We just need to point the runtime endpoint for kubernetes to the cryosocket, and our cluster should come up successfully.

B

I have a small demo showing this. Let's take a quick look, all right. I've already installed and started cryo on my system. We can check this by using the system, ctl status command and, as we see here, cryo is up and running. Now I have a simple script that I use to bring the cluster up using cryo.

B

So the main thing here is that I'm setting container runtime endpoint to the cryo socket and the c group driver to system b. Everything else is just a few configurations that I would like, for example, the ip stuff for networking, and I would like to get a dashboard and stuff off all right. So let's just call the script to get the cluster going.

B

This will take a few seconds to start up. Oh and we're using the local lab cluster um that you get from the kubernetes repo, we're just changing a few configurations and pointing the endpoint to cryo.

B

All right, our cluster is up and running now, let's grab the cube config and take a look at the cluster all right. So let's export the cube config here, let's take a look at the node and, as you can see, the node is up and running. Let's run a simple pod, I'm just going to call it mypod.

B

So it's creating my pod. Let's look at the status of the pod and, as we can see, the pod is up and running and my cluster is running with cryo successfully. We can also use the crystal tool to look at the containers created by cryo. So if I do pseudo price ctops I'll see that these are all the containers running. Currently I run the fedora container.

B

So, as you can see that that's the first one that's running, and it's that simple to run kubernetes with cryo, we can also use the similar steps to run cube adm as well as kind with cryo should be pretty straightforward and you can take a look at our installation and setup guides on the github repo. They have much more detailed instructions on how to get started with cryo and kubernetes.

D

Hello, everybody next up, I'm going to talk a little bit about cryo and cni, and some improvements we've made to their relationship, specifically some better handling around stateful pod based cni plug-ins.

D

Those of you that don't know cni is a common interface for container runtimes to create networking resources. It includes a set of commonly used plugins and defines an interface for new plugins to be created. Cryo has supported cni since its inception and cni is supported by many other container managers, but cryo really only loves cni.

D

Let's talk a little bit about that relationship, so it's age old, you know the lifetime of cryo, but even so, the relationship between kubernetes, cryo and cni is a little bit. Hacky cni is a plug-in based architecture where there's a binary on the system that cryo execs to provision the resources. This is fine and dandy for stateless plugins that are known in installation like a simple bridge plugin between hosting container that's installed through the distributions package manager.

D

What about more complex, stateful, networking network provisioning schemes or more dynamic ones where we don't really know what it is at installation time, but we do know at runtime.

D

Often, the way that more complex, cni plugins end up working is is a pod based strategy. Calico, ovn and openshift sdn all work this way. Basically, it's a privileged host network see an ipod is that scheduled onto the node. This pod is responsible for setting up some state and eventually a networking plug-in binary is put in the correct location by the pod.

D

This binary is a pass-through communication tool for the networking container itself when crowd creates a non-host network pod, it attempts to exect the binary and then the networking pod, and then that information is passed to the networking pod and the networking pod returns the information and then the crowd can continue to create the pod.

D

So, in addition to the architectural complexity, there there's a couple of pitfalls with this approach. The networking resources are provisioned by a pod. What happens before that pod is created and what happens when the node is rebooted and the pod is restarted.

D

We're going to go into a couple of improvements that cryo's made in these two areas to start off, we'll start with the former one. uh What happens when uh before that pod is created? So in this former situation, cryo would attempt to create a non-host network pod. As soon as the request came in from the cubelet on node startup, there's a stampede of pod creation requests totally unordered the cubelet's, not very good.

D

At ordering pod creation, it creates all the static pods and then it creates all the pods that it gets from the api server if the host network, if the cnipod is uh one that is kept in the api server or provision, asked to be created by the api server rather than a static pod, along with all the private network, pods that we have used to have this situation, the cubelet asked cryo to create a private network pod.

D

At the same time, cuba asked to create a host network. Pod cryo fails this first request, because the host network pod is that is the cnipod is not yet up, uh and the networking plot again is not set up.

D

Cryo then processes, the host network pod request to see an ipod takes it and you know, uh returns its creation and uh then the cubelet. You know uh re-requests the network, the private network, pod and cryo. uh Now, with the up see, an ipod is able to create the request uh complete the request, so this is okay. The pod does eventually come up, we're emphasizing the eventual part of eventual consistency, but it is wasteful. We spend this whole cycle, not creating the pod.

D

It would be better if and it is better now that we've improved it by waiting until the scene. Ipod is up so the cubelet asked to create a private network pod cryo notices that that that the cni pod is not up yet it attempts to execute the binary and the it fails for whatever reason so crow stalls the queue it's like hold on. Wait, I'm not ready yet cube. Cubelet goes forward, requesting all the pods as it does, and eventually it'll get to all the host network.

D

Pods crowd is able to provision host network pods because they don't rely on the cnipod and eventually one of those host network. Pods is going to be the cnipod and then everything goes as expected. See an ipod is created returned to the cubelet at this point, cryo realizes it can unblock all of the other uh pod pod creation requests. As uh the cn ipod is now up, so it asks for a network for all the other. Pods gets one and returns it and everything's hunky dory the next situation.

D

If you remember what happens when the node is rebooted and the pod is restarted, when a node is rebooted and the pod has restarted cryo uh so on on uh node boot, cryo first checks the disk for any containers that it used to have, and then it attempts to recreate them. Unfortunately, on a reboot, this will fail because the um all of the containers have some information and temp effects and uh that time ffs is no longer there because of reboot.

D

So the cryo cleans up. You know the information of the containers which ends up being you know the container storage, but that does isn't everything that containers provision in this case, uh if a cni pod provisioned some resources that wasn't cleaned up after the reboot, then cryo would not attempt to call cnidel the command to delete all the uh the cni state and that state would leak.

D

So this is obviously unfortunate. We don't like leaks so we've since improved cryo to now attempt to clean up the networking resources after no reboots. So we see this reboot here crow uh attempts to clean up the style pods as it did before, and this time it attempts to ask the cni plugin to clean up the resources, but the cni plug-in isn't up yet so that actually fails. So cryo will keep you trying to do that.

D

uh You know continually polling to see when the cni pot is up eventually the cubelet will, you know start up, ask to create all the pods cry. Will prioritize the host network pods and the host one of those host network. Pods will hopefully be to see an ipod. The cryo will ask the scene ipod to be created. It will be created at which point cryo will realize that the other cni resources can be cleaned up and cryo will do so.

D

So we've now shored up that leak and there's uh you know no opportunity for the cni resources. You know ipam entries or uh you know the eth uh beef uh entries. You know they they're all gonna stay around uh they're all gonna be cleaned up when they need to be all in all uh cryo.

D

By focusing its networking stack on cni uh and kubernetes is able to tailor its behavior not only to the cubelet, but also to like the idiomatic way that stateful networking provision networks, provisioning strategies, work where it's often pod, based and often um you know, behaves in these very certain ways and cryo is able to tailor its behavior so that it can account for those this makes uh cryo. This shows that cryo, along with cni and kubernetes, uh is a very good uh choice for your cri implementation in your kubernetes cluster.

D

A

I'm going to talk about a new concept that we introduced in cryo called workload types. We came across a problem that motivated us to introduce this concept. The problem was: is there a way to run all of my cluster components at this control, plane, pods and per node pods only on specific result, cpu cores, so there's a clear separation between the cluster components and the end user workloads.

A

This feature allows you to add annotations to your pods, to specify their workload type and then cryo matches on those annotations to modify those parts. These modifications are declaratively configurable in cryo.com.

A

Another use case for this feature is usernamespace support. Cryo has experimental, username space support that uses runtime classes today, but it fits more cleanly into the workload types concept and we are working on refactoring it over. Let us walk through an example of configuring workload. Types.

A

This slide has a snippet that one can drop into a file under hccryo, try.com.d or directly add to cryo.com to configure a workload type that we call pend. The activation annotation is the annotation that needs to be added to a pod to activate this workload. Type, the resources section has resources that are modified for the pods that match this workload type. In this example, the pods are getting the cpu set modified to run only on cpus, 0 and 1.. The annotation prefix can be used to override the default configured resource settings from the pod.

A

Now, let's move on to a demo of this feature, so I'm going to demo the workload types feature here, uh so I have a drop in here under etsy cryo cryo.con.t that uses the same example from the slides. We have the activation annotation, the annotation prefix and the cpu set that we're trying to pin to.

A

I have a local up cluster with kubernetes and cryo 122 running here. We will run three pods here, one without any of the annotations and the other two demonstrating workload. Types.

A

So I launched a simple httpd pod, we'll wait for it to run it's running now we will use cryctlps to get the id of the container and then we will examine the cpu set of that container.

A

So you can see that nothing is set here. It means that this pod can run on any of the cpus on this machine. So next.

A

We're going to run this pod, which has the activation annotation.

A

It's running as well.

A

And we see here that it gets pinned to 0 and 1, as we declared in the configuration file, and the final example that we have here is for a pod that does an override of the cpu set.

A

So it uses the annotation prefix, along with the container name, and we say here that we want to change the cpu set to two and three. Instead of the zero and one in the default.com.

A

A

And we successfully overrode and see that this particular part is running on two and three, so this shows the workload type feature uh in practice and will try to support more features for the resources. Please let us know what more you would like to see added under resources.

C

Let's speak about the metrics enhancements we added to cryo122.

C

The first thing, I'd like to point out is that we added a new configuration option called matrix collectors which allows us to enable or disable certain metrics on runtime by default. All metrics are enabled, but this option gives administrators a more fine-grained control over how they want to actually collect metrics and which they don't want to collect.

C

We also secure the metrics endpoint via tls, and this includes certificate creation and also validation of a certificate. This means, if a certificate is not available on disk cryo is able to create it automatically and later on. We can pick it up and exchange it, and the third point is that we are also able to rotate those cls certificates.

C

So if anything changes on this like the key or the certificate itself, then cryo will pick up those changes and will solve those certificates in case of any error. Cryo will just report that to the user and will nothing do at all.

C

Let me demonstrate that to you so to be able to run this demo, we have to ensure that we are running cryo122 and we can verify that by running pseudocryo minus minus version, and here we can see that we are on 122.0 and that should be allow us to enable or disable the new configuration option.

C

Since cryo122. We are now able to enable or disable certain metrics. So this means everyone cryo and look for the cli arguments. Then we can see new option matrix collectors which are defaults to all metrics being enabled.

C

So now, let's just limit the metrics to enable the image, pull successes for cryo and if we run cryo with enable metrics and only selecting this single matrix collector, then we can see that the cryolooks already indicate that we are skipping a bunch of metrics and that we are also serving those metrics. But we have enabled the actual metric we want to have.

C

If we now put a container image, for example the fedora minimal image, then this will take a certain amount of time.

C

But in the end we can now check for our matrix and then we grab for cryo-image pulse, and then we can see that we successfully downloaded the image and that the image counter is correctly at one now: let's clean up cryo again cryo's metrics endpoint can also be secured by using tls.

C

So if we now choose a valid certification key on disk, then we can specify that by minus minus matrix so and minus minus matrix key, and if we did that, then we can see that cryo is now matrix serving the metrics via https and to look for our metrics. We can just use curl and we have to skip the certificate because it's just this assigned one and then we can see that we get all the metrics.

C

So if a certificate doesn't exist at all, then cryo will automatically generate the default key pair for us. So if we, for example, define a metric certificating key which doesn't exist on the local disk, then we can see that cryo is now creating the certificates for us, and that is also watching those files for changes.

C

This means, if we now modify this file. For example, we put an invalid certificate into this into the source.pamp.

C

Then we can see that cryo picks those changes up, but it tries to validate the certificate, and this fails in our case, so it will not do anything at all.

C

There are also a couple of new metrics which have been added to cryo. Every metric contains the prefix container on them cryo, which allows us to easily identify the metrics served by the container runtime. The first thing we have to add is the container's om total, which gives us the total amount of containers killed because they run out of memory.

C

The second metric we have are the containers om, which contains the outer memory killed containers by their name label. We also have a metric for d-functional processes. So now we can identify zombie processes on a node and beside that. We also have the image pulse layout size, histogram metric, which gives off the bytes transferred by cryo image pulse per layer, iso histogram.

C

I would like to demonstrate that to you. So the first thing we have to do is we have to start cryo by enabling the matrix we can do that by running through the cryo minus minus enable matrix, and here in the logs, we can see that we all have all our matrix a name now I'd like to show you how cryo can count containers which get killed by out of memory. So for that we have to run a special container and we set the linux resources, memory limit and bytes to 25 megabyte.

C

We prepared a special container image for that and if we run the container in the sandbox we prepared locally on this, then we can see that cryo will run the container successfully.

C

And if this is done, then the container will be get killed immediately after startup. So if we run through the crazy tlps and list our containers, then we see that this out of memory container already got exited.

C

So the metrics now indicate that the container has been killed because of an out of memory event. And if we grab for our local metrics and look for the container's om metric, then we can see that we have now one container out of memory total of one.

C

So cryo is also able to count the number of zombie processes on a node and to demonstrate that we actually have to spawn a bunch of zombies. But for that we have to create a little helper tool how to create a zombie process. So we can just run something like exit command, ls and then start the command. And after the start, if you don't call cmd.wait, then the process will reside in zombie state, and this is what we do in this program.

C

And if we want a little tool, then we can see that the cryometrics now indicate that we have a d-functional process. Cryo requests the amount of d-functional processes on a node directly when we request the metrics. So this means that we also get a warning that we have now a d-functional process ls on the local system, and I prepared that so we have two of them, so we can distinguish them by pit.

C

This means that we have now two d-functional processes on the node and cryo is able to collect them and cryo also features the new histogram matrix for the transfer bytes during image voice. So, for example, if you pull an image which contains many different layer sizes, then this usually takes a certain amount of time, but cryo is able to lock the percentage, for example, for pulled container images.

C

We can see that we are almost done here and then cryo applies the images to the local system, and if this is done, then we can investigate actually put by spider summary count and buckets. Now. If we graph for the image plus layer size metric, then we see that we have on one hand the summary of overall bytes which have been transferred.

C

We have the amount of layers which are 20 here and we have a distinguished buckets for the histogram, so we have different layer sizes which are lower than a certain amount of bytes, and then the histogram will show up how these bytes got transferred over the overall medium.

B

That is what we had on what's new in cryo, we continue to make improvements and we'll have more updates for you. The next time we meet check out our installation and setup guides on our github and get started today. Issues and pr's are always welcome for any for from any and everyone we can also. You can also find us in the cryo channel on the kubernetes slack, and now we can take any questions you have. Thank you.