Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 4 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: rkt Deep Dive – Alban Crequy & Iago López Galeiras, Kinvolk (Intermediate Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

rkt Deep Dive – Alban Crequy & Iago López Galeiras, Kinvolk (Intermediate Skill Level)

This rkt deep dive session will provide a detailed overview of the technical aspects of rkt from its core contributors. We'll explore exactly what is meant when we say rkt is secure, efficient and composable and compare and contrast it to other container runtimes. It will also touch on rktlet, the Kubernetes CRI implementation of rkt and some of the challenges in its implementation.

About Iago
Iago brought his relaxed Spanish demeanor to Berlin a few years back. Sincenthen, he’s been diving and swimming around the internals of various Linuxnflavors; Android, embedded and Cloud. Container technologies are his currentnfocus; specifically on the rkt project where he’s a core contributor andnmaintainer. Although he once got distracted by functional programming, hisndaily tasks see him working mainly in Go and C.

About Alban
Originally from France, Alban currently lives in Berlin where he is a CTO & co-founder at Kinvolk, a software engineering team focused on building foundational Linux technologies for the cloud. He is a contributor to rkt, a container runtime for Linux, Weave Scope, a container visualization & monitoring tool, and is actively working on BPF-related projects. Before falling into containers, Alban worked on various projects core to modern Linux; kernel IPC and storage, dbus performance and security, etc. His current technical interests revolve around networking, security, systemd and containers at the lower-levels of the system. Alban previously gave talks at several conferences including FOSDEM, Linux Plumbers, IO Visor Summit, Kubecon and LinuxCons/OSSummit.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So welcome to this deep dive session. Well, first of all, we'll present ourselves so yeah Iago I'm, the tech lead at kinfolk, and this is album. You.

B

Know I'm Aldo CG, what's involved yeah.

A

So we've worked a lot in rocket I mean on rocket with Korres with the chorus team when it started and yeah so yeah. By the way we are the cofounders of kinfolk, along with Chris who's sitting there and yeah. As I said, we work a lot on rocket so yeah. We were asked to do this deep dive session, explaining some yeah technical details about rocket and rocket Linda source.

A

These these are give hub handles and Twitter and I wasn't really sure what this session should be, because in the description it should be some kind of technical discussion with the audience and it should involve yeah people experience with the project. But what I came up with on this talk is yeah I just want to discuss some interesting topics on rocket and containers and I drew some diagrams and yeah. We can just talk about those and maybe do some demos, so people can understand how the things work and yeah.

A

So that's basically, what's gonna happen in this talk. So first you're.

B

Gonna do some questions any of you. Where are the intro this morning, I woke at intro.

A

B

A

Very not very overlapping, good yeah, so how many of you tried rocket? Ok, how many of you like it: okay, yeah! So the first topic that I selected is C groups so yeah. Maybe this is too small. Does this work actually No.

A

A

So this is the C group tree for a rocket container, and you can see some things are in green and something starting red. So, for example, on the first diagram, that's the first two are in green and the other sardine breath. If somebody is colorblind or something yeah so in rocket, what we do is we mount the C group tree read-only, except the parts that actually are needed for the container to set up it, hit their own limits so rocket.

A

There's this concept of Sage's state zero is rocket, I mean it's the one that fetches the image and renders it to disk and stage one actually executes the big container and that's usually system D. So what we wanted with rocket is to let system D. Do the C group handling? We didn't want to write the see groups ourselves. So that's why we wanted to give the container some read/write part of the C group tree.

A

So before 123, as you see in the part below there were more things that were read read-only and this is actually what we implemented. So I found out, like the other day that this has been changed and now it's more things that I read right, and that was because they were experimenting with using run, see on stage two and run C requires more I, mean more access to the system to set the limits that yeah.

C

A

Are needed anyway, so what we do here is we first have everything retried, because by default system, VN spawn, which is the engine that starts containers in rocket, doesn't do anything special there. It sets up everything, sorry read-only, that's what that's what I wanted to say. It sets up everything read-only, so we this didn't work for us. So what we did was we pre mount the C group tree before n spawn and then n spawn sees that is already mounted and it just continues.

A

So in do that we mount some special knobs read/write and then we mount the parents read-only, and this worked fine and we can actually maybe see it in action. So I can start a container yeah. That's our just a simple container. For now it's just a CD and yeah there. It is yeah. Actually, let's set some limit.

A

Maybe 100 megabytes should be enough and CPU I don't know 500 million cores. Okay. So now the container is running and those limits are applied. So, let's check if they are actually applied so.

A

This is the tree of processes, and this is my container, so you can see that system is running inside and this is just an eternally, although it's a bit wrapped- and this is the HDD process. So if we go to the HDD process.

A

Yeah, you cannot do that, but let's be rude: yes, security, okay, so this will access the root filesystem as EDD views it, and we can see here. This is FSC group and memory and all the containers in racket are in the machine, slides and, let's specify otherwise or started by a system the unit- and this is the scope, so we go in here and we can go to the system slides and the HDD service.

A

So this is the HDD service, but actually inside the container, because, as I said system D is running inside the container too, so we have some limit which is close to 100 megabytes. So this should work. So one thing that we can check about those mounds is if we go back to yeah Prague.

A

Was it this one if we check the mount info file in the proc file system? This shows all the mounts that are visible here and.

A

Yeah yeah all right, so actually we don't expose directly sisyphus to the containers. So we would have to go to the PID of system D inside the container to see those because we create a new mount namespace for each app.

A

So we try and we can try to figure out which PID is this. We go there and we check mount info again.

A

And here we can see some C groups.

A

Ok, let's see ok, so this C group, for example. This is the freezer C group and it's read-only, as you can see here, but just below that you can see that this sub path, it's actually mount read/write, so yeah.

A

Yet the interesting thing here is that we pre mount the sea groups in rocket and then, since indian spawn respect those mounts and then things work as expected. Yeah. I guess that's! That's! Okay! For c groups how.

B

Many of you are familiar with looking in 2 months on c groups. In that way, not.

A

Many people, maybe.

B

20,000, ok.

A

Yes, sorry, if I'm going too fast or if there's something you don't understand, please ask because yeah, maybe it's too too much for for this but yeah. The main point is that we try to restrict what we mount as as much as possible.

A

Yeah another maybe simpler topic. Maybe this should be the first issue up in the first is: how runs OCI images so rocket doesn't have LCI support, OCI native support, so what we do is we use this component called dr. to a CI, that's used as a library to actually fetch the docker image, convert it to a CI and then pass it to rocket. So this is just a diagram that shows how this works.

A

So, first, a user will just run rocket, run docker and then some image rocket actually will actually call they convert remote repo function in the doctor to a CI library, and that has implemented OCIE client, our client that goes to the registry fetches. All the layers, then squashes all the layers together in one and then saves that to attempt directory which will and then passes a path down to rocket so yeah. This is how all CI images work and you can actually run o CI in a standalone mode. So we can try. Sorry.

A

Yeah to get the busybox image and if network is okay, it should download this a CI. And now you can use I, don't know if I still have this yeah.

A

So what we do is we convert all the all the values in the docker image to the equivalent to a CI in a CIS yeah. So you can see here. I mean this is a pretty simple image. It has maybe a special path and it executes SH and we also do annotation. So you know what sha of docker was converted, so you can retrace your back your steps to the actual image.

A

A

Okay, so in the beginning there was darker- and there was no specification of how images should be so when Korres created rocket, they decided to make an effort of standardizing this format that they will use for rocket. So they started writing a spec and then rocket would be the first implementation of that spec and yeah. They tried to learn by from Dockers mistakes, so they well or what they perceive as mistakes, so they they, for example, implemented a distributed discovery mechanism.

A

So, instead of having a central registry, you just use DNS to actually query web page on the Internet and then that webpage has some metadata that that leads to the actual image itself. So things like that, so it's I would say completely different in I mean in the form it is I mean that the data itself is basically the same right. You need to know what binary you have to execute. You need to know. You know what memory limits you have to place and all that stuff, but it's totally different from docker.

A

We don't have any concept of layers that are stacked on each other. We have a dependency concept, so you can depend from different images, so it's basically totally different. But after that docker said okay, we should standardize this. This format that we are using and and then they, the OCI formats appeared and they donated that to the community.

A

So the the thing is, the OCI format was basically Dockers format but just put in a text file, so I think that's a good thing, because it's better to have a spec than to have to read the code and see how this works, but yeah I, don't know it's still different, ACI noci and yeah. Where was I I, don't know what what else was I was gonna, say: I, don't know.

A

If I read to you a question actually yeah, it's just a some different format, but in the end the goal is the same: to specify how a container should be run.

A

Okay, yeah, so OCI in a sense, is a bit simpler because it just defines a tarball which has a route of factory and a manifest and just a simple, manifest J in Jason. So the idea was to make it content addressable from the beginning. So in darker, for example, the beginning there was no content addressability and now they added that on top.

A

So you can refer to an image by the hash, but in any case with a CIS you just have that image and that image can have a parent image and yeah that's how it works. So in docker does this concept of storage, layer, storage, drivers and yeah? Basically, it will use a copy-on-write system to actually store the layers, and then you only start the difference from the yet that layer to the previous layer, yeah but I, don't know if that explains much about why it shouldn't be I. Think.

B

The sei image format or the hacia image format does not have much link with how its taught on disk with a possible store.

B

In Ostia, well, there is OCIO bundle specification or we started the, and there is the OCI image format which are different, and it's converted on the fly by the one-time yeah.

B

The found part of the specs yeah.

A

Yeah Alban is doing that. So that's why he knows more about this yeah.

A

Yeah so and yeah. Oh sorry, what was the other speaker yeah? Tell me any other question or remark.

A

A

Yeah, so what we do is we use overlay, FS so yeah. Basically, the image is the lower part of the overlay FS and yeah. So we decided to do overlay, FS or just playing copying all the data yeah.

A

C

B

Will repeat a command to explain the how its drawn this come with us? If I summarized correctly, yeah.

A

And the tree story- I guess so.

A

Yes, I said it was its content-addressable, so each image has a hash and it and things are stored with a hatch. So here is the directory where the images are stored and you can see sorry.

A

Yeah, for example, here there's an image.

A

What is this actually.

A

Yes, that this is one of the images and, as I said, it has a root of s rectory, and then it has a manifest so okay, this is an image right, but I said that you could have dependencies. So how does this work, because the dependency can overwrite some files from the parent?

A

So the way this works is the image gets rendered on the file on the file system and in in the order of you, know the the parent first and then all the children, and then it gets rendered to this tree store, which I don't have permission to do enter, and this actually this directory actually has you know the final file system that the container will run from.

A

So all the files that are overwritten in the children will be overridden and there's some yes semi complex rules too on how to do that, because you also have you know with dependencies. You can have multiple inheritance, so there's some order that you have to follow to actually render and that's well specified too, but yeah I. Think Simone did a lot of interesting work with this dependency concept, which we didn't expect. So if you wanna come and explain something.

A

Yes, that's among what.

C

I'm Simon from working at the Recker, caring company and we've been using rockheads, it's 2015, February, 20 I think it's in production since November 2015.

C

Basically, it's the only container when time we use and we run everything in containers and our approach was to use this concept of dependency to to build all the different images. So the idea is like when you want to build a new container image which say: well, we depend on the image given in this version and you can actually make evolve the dependency independently from what's above the dependency like.

C

If you have your code in some image and I, don't know the the runtime in another image can update the runtime without rebuilding the the part of the image where your code is, for instance, I, think, that's, that's a different approach, the the good things like, as, as you explained, that everything is rendered during the runtime, and you don't have multiple overlays that you have to go through when accessing a file in a running container.

C

So you everything is rendered or the image at downloading extracted and and squashing the same in the same directory and and then you just add this overlay with the copy-on-write of this specific container and that's it.

C

We also, it also uses the same rendered container if, if you run several times the same container of the same on the same machine, so it's also depreciating things.

A

A

Yep so yeah, maybe I can go to the next one. Yeah I don't know. Actually, when is this supposed to end okay, so we have like 14 minutes yeah.

A

Another thing I wanted to talk about is all the flags of rocket or almost all the flags of rocket have a translation to assist in the unit file directives, so yeah as a set system is running inside a container, so we create a service file per up in the pot and that service file has all kinds of directories usually related to security or the environment that the app is running on. So I can show this, maybe on this container even but we can make another bit more interesting run of the container.

B

A

So yep here did I'm just mounting a device inside a container, so it shows how yeah how access to the device is restricted. So yeah, def mem is kind of a dangerous device because it has access to all the memory of the system.

A

But if you really want to mount it, you can you just do a volume yeah, it's a volume of kind host and you mount it in this target. Slash man and yeah things will run, hopefully yeah and now I wanna go to the service file. Definition to show you all these directories that I was talking about before so yeah. That's fine again, the PID of the process.

A

So this is systemd inside the container again and now we can go to its root filesystem and as any system this system. This has unit files in live system, the system, for example, and yeah. We can find the HDD service, which is here, and we can just explore it so yeah. It has some dependencies because we have some services that run before to prepare the app to actually run and some some services are run after to actually collect the exit status and write it in in the right place.

A

But here in the service is the interesting part, so yeah. Of course we need some kind of yeah executable. So this is the binary. That's like execute, execute it sorry and yeah. We also said the root directory, and this is how we isolate apps inside the pot. So we said the root directory, so the the root that the abscess is just whatever it's in its image, and here we have, for example, if the first security-related option, which is the capabilities that we give to the container.

A

So we we just chose the same capabilities that doctor gives by default and, of course, you can customize this with flags, but these are the capabilities that are set by default. The next directive is a system called filter. So have you heard of sack comp yeah? Ok, so this is the system the way to express second rules. So we have this really long list of accepted system calls that you can that the container can call. This is also adapted from docker I.

A

Think just result is this list by testing a lot of images very painfully so yeah we have a lot of Cisco's I mean Linux, has plenty of sales calls and normal apps need really a lot.

A

Yeah can go on yeah there we go yeah. Another thing we have is no new privileges, so this is to avoid applications to gain privileges via, for example, executing a set UID program, and this is set to false by default because it interferes with sacrum sorry. Anybody has some water, okay,.

A

B

All this properties in system definer is converted from docker. Sorry forget to system D. That's because we use the system is stage one, but if you use different stage, one that will may not be done with system D, but with something else right.

A

Yeah good remark, so this the fact that system is running inside the stage. One is just some decision we made, but the interfaces in rocket allow you to do any kind of stage ones, so you could run I, don't know our unit, maybe to supervise your processes or something completely different or just execute. The minor e directly like doctor does for containers yeah so and and the next one we have some options to restrict the access to dangerous files of the container. So, for example, yeah so km, is the kernel memory memory.

A

So you don't want your containers to read the kernel memory or right, that's even worse or yeah, there's other stuff. That's only read-only like this file.

B

To implement that system, D will do bind months from empty files or maybe /dev start new into this path, so that the container will not be able to access them.

A

Yeah, so we limit access to a lot of a lot of files and yeah, and this is this is how we restrict devices.

A

So if you don't restrict anything in the container, if you run as root in a container, you could create a device by running MK node and if you know the major and minor number, maybe you can create some dangerous devices so to restrict that there's this device C group, where you can, you can say which devices are allowed to be created in the C group or to be read from or to be written on. So in this case I.

A

If you remember, when I started, the container I mounted that mem inside the container, so this is what it gets, translates to translated to so device policy closed its assistant, Lee directive, that only you know, makes only allows not dangerous devices like death, no or their Falls. Their false is always going to return false. So it's not really dangerous and there's a couple of those more and then closed means that if you allow explicitly some devices, those will be yeah available.

A

So to do this, actually we need to do some trick because you remember the why we need to do this.

A

Yeah I, think system D expects the expects devices to be in /dev and when you mount a device inside the container, it's not in under slash Davis under slash bar live rocket. They whatever that. So we need to do this. This hack or something like that. I think it was like that, but yeah we just created this this temporary device, and then we mount it again where the container expects it yeah yeah.

A

So there's more options that we will have if we add added more when starting the container, but that's basically the idea, and yet you can just start your container and do this trick I do of going to the PID slash root and then explore the filesystem and and figure out how we do things so I have here a list of flags, so we have CPU CPU shares yeah except my Murray limits.

A

Yeah, this weird thing: I, don't really know what it is seccomp we can make the root FS read-only. If you want containers that cannot be changed, and this has a translation to system v2, rabid users.

A

If you want to use user namespaces, this is actually an option to assistant near spawn, not to not assist me and yeah that stuff yeah I, don't know if I have time to explain as you notify and it's kind of hard to explain, but I can show you just a demo so as they notify, as you know, device use case is to signal the readiness of a container.

A

So if you want to have like two service files and that one is dependent on the other- and you have to you- want to wait until the first one is ready to start the second one, you cannot just use run rocket directly in the system service, because when rocket will execute correctly the system the service will be marked as active and then your second container will start and but it won't be ready. Yet the first one. So this is a mechanism to mark the container is ready whenever you want.

A

Instead of whenever start, and to do this, we have to do several yeah. It's kind of a deep tree, so we have to pass the the notification message from the application to system inside the container from citizen inside the container to in spawn and from n spawn to distantly on the host. So I'll show a small demo to explain how this works. I think I have it here. Okay and the way this works in your final application is as simple as this.

A

Yeah, so you need to send a message that says: ready equals one to two system: Li and once that's happened, then yeah it means your application is ready. So we do this after listening on port 5000, and this is just something that replies hello so that that point is yeah. So this is the the script that builds the ACI and yeah. We need to add an a special annotation that tells rocket hey this support system. The notify so just pass the right variables along so I can just build this.

B

In one SD, notify is a general mechanism in system D that could be used for containers like you are showing here, but it.

A

B

Be used for any system, D services, yeah.

A

B

What communication protocol between any application running a system is services with a system D so that system D knows when the application is ready exactly.

A

So I have a service here that starts truck it and that has a type notify.

A

So if I start this service, then well, if, if you saw here, I wait 30 seconds before marking it ready to the most radius effect, so I'm gonna start this.

A

And you see that it doesn't return right away and that's because systemd knows that it's not ready. So if, in another term you know I do status, you can see that the container is activating, so anything dependent on this will not start until it's activated but yeah. Well, let me clean containers, but the pot is still running and now that it finished ya know it now it works.

A

So now I can do this. Kuroh, 5000 and I can say: yeah see the hello message. So if some unit was depending on this, it will start after there already is signaled, yeah and yeah this this. The implementation of this I don't have time to explain actually thing it's yeah time so yeah. Thank you.