Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: containerd Deep Dive – Stephen Day, Docker (Intermediate Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

containerd Deep Dive – Stephen Day, Docker (Intermediate Skill Level)

Containerd is the core container runtime used in Docker to execute containers and distribute images. It was designed from the ground up to support the OCI image and runtime specifications. The design of containerd is carefully crafted to fit the use cases of modern container orchestrators like Kubernetes and Swarm. In this talk, we dive into design decisions that help containerd meet a diverse set of requirements for a growing container world. Developing an understanding of the decoupled components will provide attendees a grasp where they can leverage functionality in their platforms. By slicing the components of a container runtime into the right pieces, integrators can choose only what they need.

About Stephen
Stephen Day is a software engineer at Docker. His many contributions to Docker ecosystem projects include SwarmKit and the version 2 specification for the Docker Registry HTTP API, and evolving the available models for container image distribution. He currently works on containerd and as a maintainer of the OCI Image Specification.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

All right, I think we're at the time now, thanks for everybody coming on a Friday afternoon at the end of the conference, really appreciate it and I know we're all tired and we've seen a lot of stuff, but this should be a fun talk. So this is the container D deep dive, I'm Steven day I work on as a maintainer of container D, among other things today, so so, actually, first of all, I want to pull the room and see how many people went to the intro talk.

A

Okay, so that's that's great, so these are the same slides as the intro talk, but I'm gonna do an actual, deep dive and do some demos and and so I'm gonna I'm gonna buzz through things. If it's confusing we'll also have a little time for questions at the end, so we should hopefully we'll have time for everything.

A

So a brief history of container D. It there's a timeline going up to about March 2017 when the container D project became a CN CF project. Ever since then, we've been working towards we've been working towards one 21.0, which was released in December of this year, and the goal was to actually spin out a lot of dockers internals into a complete project, so that docker could use it and other other things that need container runtimes could use it, such as like kubernetes and in builders and all sorts of stuff.

A

If you want to get any information about this project, you can go to the github project. Its container D, slash, container D watch out. You can end up if you're googling make sure you go to the container D, slash container d12 to file bugs and whatnot. So let me so. The the technical goals of this project were to ensure that we had a really clean, G, RPC based API and client library.

A

We wanted everything to be full o CI support, so both image, spec and runtime spec and the distribution spec, and we wanted to focus on stability and performance with a really tight core of container functionality. We also wanted things to be very disk coupled decoupled, so that the image the filesystem and the runtime could be plugged and reused, and you could kind of use it in an ala carte fashion. So we you want to be able to use own. What's required, I'll explain a little bit about that later.

A

It also has like runtime agility so that we we actually pass through configurations in container D, so, like container D, doesn't really know anything about run C. All it does is get called a binary and handed a configuration, and that's all it knows, and so what this means is you can drop in like run c or c run or rail, car or cotta containers or even g visor, that which was announced something like 48 hours ago, so that all works, because we've we've passed everything through.

A

So some of the use cases for container d, our container d or sorry container API implementations. So this would be like docker, Mobe, kubernetes, CRI, alibaba pouch in a swarm kit and then also for building images. So build kits are a good example of that. So build kit uses container d to implement kind of the container d, the container file systems and runtimes to efficiently build images and then also like IBM cloud is using container d as well.

A

It's also used within Linux kit to build the the system services inside of the container, so you can run a like. So basically, instead of having like an like an init, you can run everything as a container and and and that that's how it's built if you're curious about that check out linux kit. So a little bit about the architecture, the the main interaction goes to the G RPC API, and then we have a Prometheus metrics API, which users interact with and then in the middle.

A

We have all of these these services and they and they implement different parts of what makes up containers. So on the on the left. Here we have the storage engine, which consists of a content store which holds things like layers and manifests and images and in the snapshot store which, where the snapshot er, which does the like layered file system, support and then there's a diff service which can lets you diff arbitrary snapshots, so that you can actually build images from.

A

However, you want and then in the middle we have the metadata store which and this pins everything so the objects that you're, probably familiar with from docker or other container solutions. This is this is where that is, and they kind reached out across the system and define which so so an image will say. Oh I used this snapshot in this content, and then containers can be created from images and they'll.

A

Also reference like the snapshots that are that are doing dealing with the actual running the actual containers that you run, then the the the tasks service is actually what runs containers. So we split this out from what was in docker into so.

A

You have like a metadata thing that says: here's my container, here's all the resources that use it and then here's the runtime aspects of it, and so it means that you don't have to keep writing state into your container and removes a lot of locks, and so it's capable of much higher performance when, when running containers, then all this sits on top of an operating system in runtime. We don't really. We try not to abstract any of this, so that you can do whatever you want. So a little bit more kind of stepping back.

A

This is this is how container D would integrate into a system like mobi or CRI. Container T is actually merged, with container D now and I'll get into that later, and so your API clients, what's called like a fat client, and it contains a lot of functionality, and this allows you to modify it and change it for four different applications. So then it interacts with the container D API and then also directly with the operating system like storage file systems, networking and then the container D container D will actually control the runtimes.

A

So if you want to so I'm gonna talk, I'm gonna show some demos and show a little code from the getting started guide, but here's the link for the getting started, guide, I highly suggest you go through it. It'll show you how the API works. You can run your own containers, build your own little tiny container driver and then you should be able to modify it and maybe use it for something interesting. Then I also suggest taking a peek at the go doc, so so just to show how these services work together.

A

So the goal when you pull an image, you're looking to get a container root filesystem now what in in so what you do is you have this like this set of layers right and you need to turn them into something? And- and this is what the services do for you.

A

So when you're pulling you actually it's on the bottom here we have each service that interacts with with the with the pole functionality, and so we actually start we fetched from the remotes and then we register all the content in there and that's that gets securely hashed and then that is then read.

A

Then that registers a pointer inside of the image metadata store, which references it by its it's, what's called the digester, it's actually it's like a shot, 256 hash and then the unpacked and packed phase will pull all the content out of the content store and then will actually unpack it into a snapshot and and when it, when it does, the snapshots it'll actually like unpack a layer and then commit it back in and then it'll start a new transaction and unpack layer and then commit it back in and so on and so forth and then eventually up the other end you get where called mounts and those are actually taken in by the runtime.

A

So so snapchatters are how you build these container root file systems, I'm not going to go too much into this, but the the general. The general storage architecture is you: have this metadata store which references everything?

A

So this is images and containers and then their reference to cost the content store in the snapshot and then what you can do from this is you can get a config which is like a no CI config and then a root FS for the mounts, so I'm gonna, I'm gonna skip this part of the demo, and actually you know what I'm gonna I'm gonna skip ahead.

A

Cuz I have a really cool demo that I want to show and I and I'll come back to these slides and just just just it's more, it's more there's a deep dive talk, and so I want to get that so so so sitting here, I I have a completely fresh version of container D and so believe, I just RM dash, RF it and there's nothing in it. You can see so so container D is a namespace, so I can list all of the names faces out. There's nothing in here.

A

You can see that this is just a completely untouched container D. So we're gonna start out and we can just do normal things with container D, so we can like pull an image. Hopefully, I have a pool command here we can just do Alpine lattice, and so this just works very, very familiar interface and then now we have an Alpine image and we can just we can list like anything. This is just this is boring stuff. This is what is how we want it to be really boring.

A

We can run that so I can I can say you know just CTR run latest and give it like that and then give it a you know. This is a demo. This is not a test, you know, and then we can, you know, show a now that that's a bug, that's also a bug so anyway, so so we're running containers here. This is very serious, stuff right, we're running containers it in this is, and so we can we can. We can actually see here the get another tab there. We go.

A

There there typos mistakes, so so what so? What we can see here is is from so I can do containers LS and we can see you know we can see my o CC containers there we go so we can see the containers that I created right, and these are just metadata objects, so I can do container info demo too, and I can see like up here's the snapshot key we're using overlay. This is the created at an updated time in here is it'll. Show you a second. We can see the run, see spec.

A

So this is this whole blob here is the run. See spec and I'll. Show you how to unpack that in a second, and we can also see like oh look at this. It's using this image and it's using this runtime and you can kind of dive down it gets so we can. We can actually pipe this to JQ. It's a really cool, little tool here and then pipe this through here and we can unpack.

A

This is all base64 encoded for fun, and then we can see here and what we have here is just a good ol. Like this is a run, see spec right here so container D is not actually abstracting. This that goes right through. You can define anything that runs so you can do or in anything anything that's capable with run C. You can do right in container deep and I'll show a little bit more about that.

A

Now that so we're one thing, I will show you the snapshots before we get into the cooler part of the demo. So there's so so the snapshots are like a filesystem representation. So it's a transactional model where you can like make some changes to the file system, and then you can commit them and then you can make changes on top of that and it works just like docker have you ever played with like so when you build it.

A

When you have a docker file, it has like run and then it'll have, like it'll say, make this env and then do that and each one of those lines is a layer you this one in container D you can just have you have a transaction? You can do a whole bunch of operations and commit that whole thing and they're what's called snapshots.

A

So what we can see here is that I have demo one and it's a an active snapshot and it's parent is this cd70 blah blah blah blah that right there is the Alpine image, and then we can see our demo to as the same thing, it's probably easier to see this in a tree command. So we can see both of these two containers descend from the exact same filesystem, cool right, yeah. All right should be a little bit boring or like. If it's a little confusing, that's okay.

A

We can ask me questions later and and figure that out, but the real thing I wanted to have so, let's, let's put out of this container and I, want to show you I have I have so so we recently released container D, 1.1 and container D 1.1 has built-in CRI support, so it works directly with kubernetes and I want to show both this with this working, but I also wanted to show you how you can do some of the things that things I just showed.

A

You with with pulling out the Runcie spec you can do that all the way from top to bottom, with kubernetes and see actually the pod manifest working at the container D level. This is but first of all, we have to get there so actually I'm running on top of docker right now so and I'm gonna convert from using from running my local CRI on top of docker to container D. So just to show you I have some pods right here. You can see I've I have an engine.

A

X deployment I have like calico running, and you know all of the all of the good stuff that you would have running inside of a kubernetes cluster and just to show you this is. This is not fake, there's no recordings! If I mess this up, it's going to be very embarrassing, but hopefully it works so curl it again. Engine X welcome to engine X. This is this is great stuff.

A

So so what we're going to do is I'm gonna go in and so there's a align inside of the cube, a the cubelet service cube ADM file and it actually configures the the runtime so right now I have it is there's no arguments and then what we can see here does it need to be bigger or are we is this?

A

We all see that probably a little bit bigger, so we can see container runtime remote and then we can see, there's a request timeout and then we can see container runtime endpoint, and this is pointing at the container D socket.

A

So we're going to comment out that line, we're going to comment this line back in and then we're going to do a daemon reload on system D and then what we're going to do is we're going to take all these docker containers and we're going to start killing them because yeah that's what we're going to do. We're gonna say so: we'll go X, arcs docker stop, and so this is gonna. This is completely killing my kubernetes cluster. If I started curling right now,.

A

This is probably gonna stop. This is not going to work anymore right because we're shutting so we basically shut down the whole kubernetes cluster. If we do get pods it should probably it's probably not. It's actually started up up on container D, alright. Well, no, no, no oops!

A

Actually, no, we need to do. We just stop the cubic, because the cubelet is restarting everything yeah. So here we go. Let's stop everything again! Don't do this. This is not the right way to convert to it. I just wanted to do this in a demo, because it's fun got plenty of time. Alright. So then, if we start getting pods, we should see things falling apart, container DS not doing anything. Yet we should see that cubelet services stopped and then let's do one more docker PS and make sure so we're not running anything.

A

So we should be able to. We should be able to restart the qubit service. Now with that new configuration, and if we look at the qubit service you know logs are coming by gets her work in. We can start looking at the Oh.

A

Yep, so that so that so right now so right now the the API server isn't available. Yet because it's trying to start up, we can see here when we, so this is another tool called cry cuddle and it lets you directly interact with the seat with the CRI and I'm gonna point this at container DS socket now, and so we should start seeing these pods come up in container D.

A

Oh there they are, and so we can see that, like the controller manager and the scheduler and the API server and SCD have all started to come up, and maybe if we are lucky Oh No yeah, that is not working, and so we can look at you know we can look at journal control, don't.

B

Still not work in what aren't we missing.

A

Maybe that's not good.

A

Let's really get rid of them. Okay, so there we go okay, so it's starting to do things.

A

Okay, that's better, it looks better see, I could have come in and just had it running on container D and then you wouldn't believe you wouldn't believe me, but.

B

A

Okay, let's actually look at some more slides. While we wait for that to come back up so a little bit about running a container I, don't want to share it all right, it's a little bit about running a container in container D. So this is.

A

This is a general architecture of container DS runtime system and when I was talking about all a cart earlier, so in the so when we when we want so earlier, I talked about getting mounts out of the actual snap out of the pole system and I also talked about getting config out of that, and these both both of these are what's needed to do the actual runtime and so on the bottom.

A

We have our containers metadata service where you actually register the the runtime can figure out when you start up you hand those in to the task service and then note, and then that will all be directly passed through to the runtime and then so here we have an example of different kinds of runtimes that could be in use.

A

We have like Linux, there's windows, containers that can be used on Windows and then, as well as kata containers and I, didn't update the slide for G Weiser, but I guess you could use G advisor now to G Weiser, so this is very similar to the flow before and so on. The bottom. We have the services that are going to interact with, with with starting a container. We have the image service, the snapshot service and the container service and the task service and our we have the whole run flow.

A

So this would be the equivalent of docker run, but this CTR run we start out and we initialize a snapshot. So we we pull off a snapshot. So I showed you the tree command and then we said set that up by creating a container like a run, run, see config inside of the container service, and then we can start that and then you know out the other side. We get running containers because that's the whole goal right back to the demo. Yes,.

A

So so this so like ten minutes ago, this work this took like 20 seconds and you know: don't don't switch this in production yet, but so we should see so that so this is starting to come up. Are these backing off at all, see that curl? No, that's not working, okay, so it but okay. So what it should be so before remember, I showed you that the namespaces were empty, so we're actually gonna see we're gonna start seeing namespaces. So we have the default namespace and that's where I ran the containers before so.

A

If I just do CTR container LS right, we get that, but I can change the name space to kubernetes so like so. This lets you like logically separate everything so I can run. Kubernetes I can run docker I can run build kit, I can in docker it will actually run its system. Containers inside of inside of this and they'll all be separated. Logically, so they can run on the same host without interfering with each other.

A

So when we do, when we do the container LS command with the kubernetes namespace, you can actually see you can see that all the different containers here. So we can see like a bunch of pause containers that hold open the name space and we see like cube proxy. We can see the engine X application running, we can see, calico, etc, etc, etc. So we can actually the cool thing about container D is like it's completely debuggable from a container perspective. Indian, so I can take one of these container.

A

Ids and I can I can get the same info on it like I showed you before, and so before. I showed you the runs, C spec, but now I'm gonna show you how to get the container metadata from the the actual, the actual container. It's or sorry the pod manifest. So this is actually stored directly in container D and you can get to it and you can. You know, pull it apart in various different ways, so we this is called an extension and it's stored under a key that is unique to the CRI implementation.

A

So if I had something else, another plug in another thing that I wanted to store on there I can you can store all these different things on containers they're a little bit like labels, but they're they're, designed so that you can store like larger, larger pieces of data. Let's.

B

A

Thank you, whoops.

B

And then base64 the code that and then we'll.

A

Make that nice JQ- and you can see here this is a like pod, manifest for that thing. So now we can. We can actually someone to show you in cry cuddle and hopefully this works. I can do and inspect on the pod id that'll work inspect P, oh yeah, your.

B

Pod and cry cuddle not found, mmm let me find it for you.

A

All right: well, that's not working either, okay, so but but hopefully all of our pods are coming. Oh, my god, all our pods are up and guess what so, because the way cube ATM sets up the etsy D deployment inside of inside of the it's. It's I think it's in it's like a manifest inside of Etsy kubernetes.

A

It actually uses a mounted volume so that when you switch runtimes and anything, that's like defined outside the runtime, not using that's actually inside of the cubelet or in it's not implemented inside of the CR I will get transferred over so notice. That, like my nginx deployment, is working right and I didn't have to do any changes. So this is so you can. You can actually upgrade nodes, you can change them to container D and your applications will still still work, obviously test it and heavily.

A

If you do that, so alright, so um from there how much time? How much time do we have we about 10 minutes? Do we want to so I can show you some more code or we can take time for questions. What any dessert are there any questions at this point?

A

Now: okay, let's see code, okay, so the the code so container D has what's called this like it's a it's, it's a it's a it's a client library that does a lot for you and what that means. Is that you can you can change what the code does? We don't do like plugins, where you have like hooks that that run at various points we did what we did is said. Everything that could people would might want to change. The behavior of will actually be done in the client code.

A

So this first example is how you do pulling an image, and all you have to do is import. The client connect it to the socket and then define your namespace and then call pull, and this will actually just pull the image, and you can see here on the left of this slide. We can see that we set the namespace, we do a CTR pole and then we can actually see what the what the images are and and then here's an example of running a container. It's it's very similar.

A

We use the same client, we declare a new container, we're going to call it Redis server and when we define exactly the image that we pulled in the previous slide, and we actually say we specify how the specification is created so I can I can make my own Combinator's, that that add, maybe my own set conch filters or add special environment variables that you need, and then we actually can define how to get that that image can figure out.

A

So if I have a completely different image format that you know that's newly newly conceived I can I can define how that actually gets converted into a run, see specification or a specification for whatever runtime you want. So then, once we've created the container object. We then create a task for that container and then and and that'll start up and then we so then we specify a weight and then once we're waiting on once we have a weight on that task.

A

We start it up and that will actually run the container so once you once you're actually running the container. So this is an example of killing. The task and it'll show a little bit about more more about the weight and exit, but you can see we wait and then we can call it kill on that. And then then you can wait on this channel to get the status and then you can respond with the status and I.

A

Have this as an example here that we can just run and it will work directly so so, just just like this is very similar to what we had before so like we're connecting to the socket we pull in example, namespace we pull Redis Alpine. We have this option to unpack it into a snapshot. You, don't you, don't that's not required you can. You can not unpack it into a snapshot, because maybe you want to add your own layers in between containers or do something completely different.

A

We define a new container and then we we could. We have a few same thing on the slides before except I. Think we pull in a new snapshot on top of that and then we'll create a task just like the slides, and we will wait and then kill that task and what this so running this so I can see it in here and then and we can actually do this build and we can.

A

We can run this example and it needs to be run as sudo, because it passes off some FIFO into container D, and we can see here this. So this is I think that's pulling the image right now and then it should run Redis.

A

So, what's what's that what's happening right now, we should be able to use.

A

So we should see a new namespace for the example alright, so we see that and we can see that there should be a new container in there in that namespace.

B

A

Maybe it's just like hanging on the Wi-Fi.

A

Yeah, it's it's. It's pulled the image. Oh there, it goes Redis, okay, so yeah, so I pulled the image ready to start it up. It hits some logs and then we waited like three seconds and then we killed it and it it exited with an exit status of zero. So it's very very easy to build very small lines of code. That's that kind of code is that is the code that you would see in docker or in in CRI.

A

So the so that you can actually compose this with the other other things I'll make these slides available, but here's an example that would run H top. So you could add this add this to your run, C spec and it will run H top inside of that same container, and you can just you can build these there's things you can do with networking. You can do things with file systems, mounts TTY, etc.

A

So with that, so are there any questions at this point? It's kind of been like I want to get in a lot of demo and stuff. Like that and I know, it's been a bit of a frantic talk, but is there because seriously there's time for questions? No, all right! Oh yeah, go ahead.

A

No, no, this there's, no, the only thing I did in docker was remove all the containers from it and then run them on container D. This is completely without I mean. Let me see if I can show you so so. Here's dr. docker was is running nothing right and PS a and then we can actually so show like ps3. Maybe that might might be a good way to show it.

A

So here we see gnome terminal and we see sudo container D, which is running on this terminal right here and then you can see all that can all the container shims. So we can see Cuba, API, server, scheduler, etc. There's no docker running here any other questions, I'm. Sorry. What what about it.

A

Well so the H top is just an example think so so if you want metrics that we actually have a Prometheus endpoint and you can just curl that- and so let's see here, v1 metrics and you can just pull these into Promethea. So we have all sorts of containers.

A

So like so so here's you can go through and get the G RPC metrics, but every single so but you can get all the metrics for, like your containers, like CPU usage, you can get the the memory usage like block IO usage, any like network usage, disk usage, that kind of stuff through Prometheus. The HT top example, is just a common Eider. It's not how you'd normally get metrics for a container.

A

No, no! No! No! So this the the HT top example has nothing to do with getting metrics in a production system. Normally it's just an example of how you can do something like silly or non-standard by just adding it into your code and and using that, if you want, if you want metrics, you get, you can get them through Prometheus with container D all right. Any other questions.

A

Yes, I forgot that sorry.

A

No III chmod of the socket so.

A

Yeah: here's the here's, the cry, cuddle, pods that you can see.

B

Yeah there we go.

A

Inspect pee yeah, so this is actually. This is the same pod manifest that was sitting in container D before right, yeah. So, okay, thank you is any other questions.

A

So so you so like, if you just want like a really simple container API and that works for you, docker is great and container D is really built for when the like 20% use case, where you need something special like you need special mounts that happen at certain times or you want to integrate with some like wildly different networking system or just something you can't do with docker and isn't gonna get added to docker.

A

It's absolutely for production systems, yeah.

A

Yeah, if you need special, is so so, if you're, if you're, if you're going it with your running kubernetes like container d, is a great choice and if you just need a simple run time with the CRI support or if you need something.

A

If you want to do something really custom with containers like build an operating system or build some other sort of like a can, yeah, yeah or or that kind of thing where's like if Dockers model of containers doesn't work for you for some reason like you can go to container D and customize it and make it make it work with with you, can still use, docker images with it and and then, and you know, modify how that works in some way.

A

Let's say every single docker image you wanted to add a file at slash, hello and route like you could do that with container d, but that'd be very hard to do with docker yeah.

A

Yeah so like this, a it's so like, so so, for example, so you have CRI, which is the kubernetes container runtime interface, and then you have docker right and those are both at this kind of the same level of abstraction, but they think about containers differently, but container D is able to fit into both of those models. Does that make sense? Because it's flexible enough? It doesn't make its own abstractions they get in the way of kubernetes or docker, or build kit or or any large list of different systems.

B

A

Mean it's: it's not meant to be a replacement for docker in that sense, so, but if I suspect that it's a but if you're, if you're, doing, host mounts if you're doing if you're running things with I mean for the most part, if you're, just using docker in your development workflow continue using docker. It's it's a bit like we're. Not we're not going to try to compete with docker. That would be a silly waste of engineering, but.

A

Yeah, you need to know yeah yeah so but yeah in the future. We might have that there there might be some some asks for a more a more user, friendly, command-line client. Just just the thing. The thing is like container D has such a wide surface area that exposing all of that in a command-line client would take away from its capability over time.

A

Any other questions.

A

All right, how much time do ya? Ok, all right! All right! Thanks for coming.