Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022 [End User Sessions], 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keep Calm and Containerd On! - Anusha Ragunathan, Intuit Inc

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keep Calm and Containerd On! - Anusha Ragunathan, Intuit Inc

Letting go isn't easy! Especially when it comes to your Kubernetes cluster’s CRI implementation. Like most big Kubernetes deployments, Intuit’s 200+ clusters with 20000 nodes were running ‘dockerd’ as the CRI runtime, with dependencies on the docker API and CLI. We migrated our fleet of clusters to ‘containerd’. Whether you have a complicated Kubernetes installation with customized cluster addons or a simple set of clusters, you will be affected by the upcoming removal of dockerd from upstream Kubernetes. Come listen to us, learn from our journey and be prepared to make this migration smooth and seamless. We will share lessons learned migrating clusters to containerd. From issues faced with log management, SELinux and GPU support, to rewiring cluster addons related to CNI and runtime security, this talk is about Intuit’s journey moving to containerd. We will also talk about rollout of containerd to our production clusters and how we handled compatibility issues during cluster upgrades.

A

Welcome everyone today I'll be talking about how to keep calm and container deon. My name is anusha raghunathan and I'm a principal software engineer at intuit.

A

Introvid is a fintech company that makes software around tax preparation, accounting, consumer credit reports and stuff. So if you've ever used, turbotax mint credit, karma or quickbooks, then that's us.

A

The agenda for today is basically starting off with why. Why are we even doing this talk? What is the background? What are cris, then we'll do a section on how we went ahead, planning the migration to this new cri.

A

What happened during our great migration and some performance analysis that we did with the new cri and finally finish it off with takeaways.

A

Before we get started, I want to give a quick intro introduction about our kubernetes based infrastructure.

A

We run about 220 plus clusters and they average about 16 000 nodes, and this number of nodes actually goes up pretty high during our tax peak seasons and a number of kubernetes namespaces, roughly about 15 000 and odd.

A

We run about 2000 production services on this kubernetes-based infrastructure serving about 5000 developers, and we have about 17 000 assets that we manage with this, and each kubernetes cluster runs about 25 add-ons on top of the vanilla kubernetes cluster we get. These are primarily around security and compliance cluster life cycle management.

A

I want to call out our keiko proj, which is a intuit, uh started, open source project that manages our cluster add-ons instance management, upgrade management and stuff. Then we have add-ons around observability metrics tracing and logging, networking, cni and service, mesh storage, reliability, testing, around chaos, experiments and finally, container native workflow engines, specifically, argo workflows,.

A

Now, what's a cri, a cri is a container runtime interface and kubernetes. Has interfaces for running your storage and running your network container, networking using cni and csi similar to that for running a container and doing some image management.

A

There is a cri interface that the kubelet calls out to and a high level container runtime like docker, shim or container d manages of containers and images life cycle of them and in in in turn, calls out to a low level container runtime such as run c and a cri, is a well established uh interface that the kublet calls over and it's grpc based and examples of container runtimes, like I mentioned docker, shim, cryo and container d.

A

Now, why is it that we're talking about container d and cri and stuff at this point in time? Typically, your average kubernetes operator doesn't have to worry about these things.

A

However, now we have to worry about it because of something that happened in kubernetes 120, which is the primary container runtime that was used in kubernetes, which was docker, shim got deprecated and docker. Just a little bit of history. Docker demon was the primary runtime for kubernetes, but it was not cri compliant, mainly because docker demon was written way before kubernetes originated, so a thin shim layer was written on top of docker daemon to make it cri compliant.

A

A problem was the shim struggled to find maintainer ship and also there were rising container run times that were cri compliant, so in kubernetes, 120 docker shim was deprecated as the default uh cri and in to kubernetes upstream 124, it's going to be removed. That's why we all have to worry about the cri runtimes now, so we decided to pick container d as our cri run time. uh Why?

A

Because docker daemon eventually calls container id anyway today, even when it's not the shim, it's not the cri and it's been battle tested really well and it offers better performance in terms of cpu memory consumption as well as pod startup times, and it's supported by our cloud provider. So we went with it. It was an easy choice.

A

uh This is a quick comparison about the container runtime invocation between docker, shim and container d. You will notice that in docker shim we have the extra hops between dock kubelet and docker shim and eventually reading up to container d on the right side, you will notice that the kubelet directly calls container d and the extra hops are eliminated.

A

So how did we go about planning our migration.

A

First, you need to understand the cri wiring in your cluster. Look at your worker nodes and see how many places you have docker daemon, docker, shim sockets, being exposed to all of your cluster components in our clay. In our case, the blue docker veil in the middle is actually the heart of a lot of things and it was exposing docker, shim and docker docker shim socket as well as docker demon socket.

A

And um if you look at the bottom of this diagram, there is a box with all our add-ons that were actually relying on a cri socket, namely the cni chaos, engineering, toolkits, falco add-on, which we use for security scanning for our container runtime images as well as argo, workflow controller, the other obvious clients of docker uh daemon, where the kublet uh making the client connection through grpc, as well as docker, cli and api.

A

The settings on the left side were mainly around docker daemon configuration and, as you might see, there is sc linux policies that we were writing based on docker, daemon, gpu configurations for our ml workloads, as well as container log management and another indirect uh dependency. That we had was that we were expecting the container logs to be in json format, which was offered by docker daemon, and we were using fluently to ship them out.

A

So we had to rewire our worker nodes to use container d or other mechanisms when we had to migrate here is uh how we rewired container d.

A

So we mainly baked the container d client, which was the cry cuddle client which worked out well for us, and uh the cricket client would talk to the container d socket and um the continuity socket was also used to communicate with our add-ons in the bottom. So the cni chaos, falco plug-ins all ended up working really well with our new container id situation and notice that the container d config file now actually still continues to take gpu, config and sc linux. But the container log management has now moved to the kubelet.

A

So it's no longer maintained in container d. You just have to configure your kubelet um accordingly, also notice that our argo, workflow controller is not dependent on continuity. Socket anymore. Our workflow controller had a dependency on docker, mainly because uh it needed a primitive to share the container name spaces.

A

Two containers needed to have uh the same process uh namespace, and we were now able to use the kubelet because kubelet provides process, namespace sharing, so our dependency on the container runtime was removed, and now we had a dependency on kubelet and finally, the dependency I was talking about. As far as the json log format was that container d doesn't have a concept of logging plugins and it was only logging container logs in text format and we'll see what happens because of that in a bit.

A

A word about container declined, cry cuddle. We baked it. Basically, we baked this client, basically into all of our worker nodes and for the most part we were able to actually find parity with the docker api. Cli main handy commands that we would use for cryker ps, krakatoa images uh instead of docker inspect. If you were used to using docker inspect, you could do the same thing with krakatoa get info which was used in our use case primarily to work around sc linux policies.

A

So you can actually explore more on cry. Cuttle um as your cri continuity client, we couldn't find exact parity for some of the uh use cases. uh One thing I would like to call out is the docker system prune command that we were used to for actually cleaning up leaky containers um again, since docker was more than just a cri, he did a lot more than just container and edge management, um so assist system prune.

A

We had to find uh like some hacky bash scripting and cron jobs on top of cry cuddle and we were able to get it done. So the big takeaway is get familiar with some container d client and so that it's helpful during your migration.

A

um Another container decline, the obvious one is kublet and the one thing that was handy again for us during our migration is that kublet actually can be configured to have a different cri.

A

In this case, the options are container runtime and container runtime socket where you could actually spin up a test cluster today by setting these options- and you will get a container d cluster, so it's easy to just basically jump start on it and see which part of your bootstrap code. You need to be changing in order to actually perform your migration, and, like I mentioned earlier now, log management has moved from the cri to kubelet, so you can get familiar with the log rotation options for max size, as well as max files.

A

All right, so we planned our migration. We made all the code changes that were required in our cluster bootstrapper. We had reconfigured things to um use the container d configuration rather than docker and all our add-ons. um That code was also changed to accommodate for container d. A lot of our end user teams and our platform teams had also migrated their code path.

A

So one thing I would like to mention here is that at into it, we go through monthly cluster upgrade cycles, mainly for security and compliance reasons, as well as a time to actually introduce new features to our kubernetes platform, and this is the time when we actually migrated from docker d to container d, and one thing to note that is that we perform rolling upgrades of our clusters, which means that you started out with docker as your cri for your cluster.

A

And then you ended up with container d a cri, but then there is a period of time during the cluster upgrade where you have a mix of both. What kinds of nodes in your cluster and we guarantee to our end user developers that there will be zero downtime during the upgrade.

A

So we planned and planned and we did the right things any guesses on whether it went uh smooth or did we have any gotchas.

A

Well, of course, we had gotchas and I'm gonna be talking about uh two of them. Today. First was about the logging pipeline.

A

Now before we get into the details of the problem and how we solve for it um a quick refresher on our logging pipeline. We use fluentd as our node driver for logging and a fluent is responsible for log collection.

A

Log parsing, if it's needed, as well as shipping it to our log aggregator, which is splunk um fluently, takes care of shipping pretty much all the logs uh in our nodes, but the critical ones are container logs and we cannot afford to miss even a small gap of logs.

A

So log loss was not a is a big no-no in our infrastructure and fluvindi runs as a daemon set and it's a cluster add-on.

A

So what is the first problem? Well, our entire logging pipeline had the assumption of json formatted container logs and container d basically broke that assumption right from container logs to fluency to our to how it reaches our splunk servers. Everything was expected to be in json format, and this was because uh json offered better performance, as opposed to other formats that we had considered before and what happened was because of this change.

A

In this uh log format, there was log loss because there were portions of our pipeline that did not recognize formats and we ended up with log losses and, like I said this was a big no-no and how did we solve it?

A

Well, luckily, even though container logs were in text format there, there is a predefined specification for the format, which is a time stamp, the stream tag and the actual log message, and they were all space delimited and we could pretty much use fluency configuration to actually parse it out using a regex.

A

So that's precisely what we did, what we did was we changed our bootstrap code to make sure that each node came up with the cri runtime. That was nes. That was running that node and our fluency bootstrap container would read that file and load up the right, fluency config map for docker. It remained the same json based configurations, but for container d it was a regex parser that would extract out the log message and eventually send it out to splunk the one thing to note.

A

That is with this approach we did observe a 17 dip in performance of log throughput that was sent out, and this was because of the additional regex overhead that we were seeing with fluency.

A

What was the second problem?

A

Well, the way our cluster upgrades work is that the nodes are rotated out first and then the add-ons are upgraded and what was happening was there was a period of time when the nodes were rotated out and the new cri was coming in place, but fluency still did not recognize that they need it needed to do the bootstrapping changes we had made. As a result, there was again log loss during that period of time when the clusters were getting upgraded.

A

So how did we solve it? It was pretty simple. We just got our code for the platform, uh the release before the actual migration to container d, so that that that was one big takeaway for us is you have to actually not just make all the wiring changes you have to actually get it in a release. One release at least n minus one release before your actual cri migration.

A

The second gotcha I'm going to talk about today is regarding cni again a quick refresher, the cni that we were using along with pod and host networking stack wiring for um for canada. Networking also took care of ipam daemon. So basically, the ipam daemon was like this local host demon. That was responsible for two things: one maintaining a warm pool of ip addresses to hand out to pods and also taking care of allocating and deallocating ip addresses for the pods and for ip allocation and deallocation.

A

It would query the cri socket to get a list of running pods and cni was running as a daemon set in all our cluster nodes.

A

So what was the problem well to prepare our cni for our cr container d migration? We had actually mounted container d socket uh in the cni pod spec, as expected from our cni vendor and cni is actually a bootstrap add-on for us. So in this case the bootstrap add-ons get rotate like upgraded before the nodes get rotated out.

A

So you can see the problem right. This meant that, like a node would a docker node would still be running, but the cni would think that this was container e um node and it would query the continuity socket, get an empty list of pods and voila. It would be like okay, this node doesn't need any more ips, so let me start de-allocating from uh the docker pods, which were actually live and running, so this was again a big no-no for us.

A

So how did we solve it? Well, we created a generic sim link for whether it's a c a container d socket or a docker socket in our bootstrap code. So if a node came up with docker daemon, then this generic dot sock would actually point to that. If it came up with a container d, then it would point to canadian id dot stock and uh the cra stock was actually mounted into our pots.

A

Cni parts spec, and we made sure that again that these changes got into a release prior to the actual migration, because you had to prepare your cni a code to handle this before the actual migration.

A

All right: let's talk about performance.

A

We had already set expectations that pod startup times and cpu memory. Consumption for pods would be lesser in containery, but we wanted to actually verify that and see how much of a performance and gain we get.

A

So our setup was running. Docker, 19.3 and container d146 on kubernetes 121 and the test service is a java spring.

A

Boot application that very closely mimics our developer environments and our test line generates about 6000 transactions per second, and these transactions are mostly a combination of reads and writes into a memory heavy um like memory, heavy uh load, a graphql in memory database calls and um the test was set up to actually ramp up for the first 16 minutes and then have a steady load of about 6000 transactions per second for the for two hours and we started with an hpa uh replica of count three.

A

How did we do in this?

A

Well, uh continuity definitely fed well, especially in the startup times. We noticed that containery startup time. Maximum latency was only about 120 seconds in our synthetic workload as opposed to docker, which which took about 200 seconds and we're pretty excited about this performance game because it will come in handy during our tax peak seasons when there is very high hpa kicks in, and we have a lot of replicas running at the same time and startup times really matter and during the steady state of about 120 minutes.

A

We noticed that, because of a lower slightly lower cpu consumption, there was lesser number of pod and node usage because it like it was just a slightly more efficient and hp doesn't kick in for a slightly longer than when compared to docker.

A

So I'm going to wrap it up with some takeaways.

A

Understand your cluster cri wiring. This was our use case where we thought we could just start off with a simple migration from docker to container d. It turned out to be a several month project for us, so understand your cri wiring and plan to test co and verify ahead of time because kubernetes 124 is coming up and there will not be a docker shim anymore and the third big takeaway for us is: you have to account for live cluster migration.

A

If you have a big big platform like us for us with the 220 plus clusters we had and with zero downtime you had to, we had to make sure that all the live migration use cases were handled.

A

Thank you very much and we are hiring at introvid. So if you're interested please come and talk to me and I can take some questions now,.

B

Okay, hello: uh you mentioned that your cni needs access to the docker, socket right, yes, and do do you know if the docker socket if the container this socket is compatible like fully compatible or you need to change your cni codes to read it in a different format.

A

So the cni stock, the cni, basically queries the cri socket mainly to get a list of containers and parts so that it can actually do ip allocation. So, as far as compatibility goes, it's a it's. The interface is pretty well established and to what it is expected. So we didn't see any compatibility issues there.

A

It was more around the live migration, where our cni did not account for both set of cris to be available at a particular point in a cluster upgrade situation.

C

Just a question about the gpu you see in the description you hit some issue with gpu. Can you say more about that? Please.

A

So the gpu issue was about so it's more about a vendor issue more than a container issue, so we get our gpu support from nvidia and our cloud provider is aws and uh the issue there was the nvidia gpu operator out there for container id doesn't work out of the box for aws.

A

It's mainly around os support, so we had to work closely with aws to actually get a separate recipe and bake it into our amis and the other uh sort of glitch in there was the end user license. Agreement between nvidia and aws had to be sorted out. So so far, uh we've done some preliminary testing with gpu. There have been no issues with container d, although we haven't gone fully production with it.

A

But I think if you have nvidia a gpu operator running on centos and or ubuntu, then you have out of the box support.

A

All right, thank you. Everyone.