Rust Programming Language Rust and Tell Berlin, 9 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rust & Tell Berlin September 2021 - Michał Rostecki: Developing and managing eBPF programs with Rust

Description

No description was provided for this meeting.
If this is YOUR meeting, an easy way to fix this is to add a description to your video, wherever mtngs.io found it (probably YouTube).

A

(Closed captions, provided by prisma).

A

So a bit about me,, my pronouns are he/him or they/them.

A

I work at suse, and my background before coming to rust.

A

Was mostly python and openstack,, and afterwards golang and kubernetes., and mostly I,. Nowadays, I use rust, go,, sometimes still to c,.

A

And I still mostly work with kubernetes and container-related topics. and sometimes with linux, kernel and of course,, a lot of bpf.

A

I will start with this short introduction,. "What is ebpf?" and ebpf is a mechanism in linux kernel which allows us to inject programs.

A

Which are triggered by various events or integration points in the kernel.

A

And the,, one of the most popular triggers for ebpf programs in linux kernel is network I/o.

A

And there is, well, the most known projects,, which is using ebpf is,. It's called cilium,, is using it for example.

A

There are also a lot of projects which are using ebpf.

A

For tracing functions,, both in the kernel and in the userspace.

A

And yeah, like tracing some function, calls is also.

A

An event trigger which a bpf program can be triggered. By.

A

And the last thing I mentioned here,.

A

For me very interesting, one, are lsm, hooks, and lsm. Hooks are custom hooks in the kernel code,.

A

Which allows, allow to create a security, security system.

A

Which is denying or allowing some action.

A

and if you know, a selinux and apparmor, for example,.

A

These two security systems are, in fact using lsm hooks.

A

But nowadays, since kernel, 5.7,.

A

You can use those lsm hooks also for creating bpf programs.

A

So you can, in fact, try to create your own security systems.

A

Similar to,, let's say,, selinux or apparmor,,.

A

By writing your own bpf programs and not doing.

A

Any change to the upstream kernel source codes.

A

And bpf programs are at the end compared to the byte codes.

A

You can also write bpf assembly,, but for example, I've never done it.

A

Mostly people write bpf programs in c,.

A

And then they,, the c source code gets compiled to the bpf byte code.

A

By llvm., but nowadays there is also a similar possibility.

A

Of doing it from rust, since rust, underneath also uses llvm.

A

So at the end, llvm is still used to convert the rust code.

A

Into bpf code,, although I'll come to that later, but yeah.

A

The bpf programs are supported in rust,, but it's still super early, super experimental.

A

There is not yet any project which would be used.

A

On production,, which is developing bpf programs in rust.,.

A

Like,, the most known projects is still done with c.

A

And well,, some sorts of (inaudible) programs are like tracing events.

A

Or getting information about events. and some of them can make actual decisions.

A

What's happening with some events happening in the kernel.

A

And well, I mentioned that probably the most popular kind.

A

Of bpf programs are the network ones.

A

and in case of network bpf programs,, you receive the events from the kernel, that's,.

A

Or some packet came to the network card, or there is a packet which was parsed by the kernel and.

A

In that case,, you can make an actual decision,.

A

Whether you allow that packet to come,.

A

Whether you redirect it somewhere else to some other network interface, or maybe you just want to deny it and it's not come today,.

A

Down the, go further anymore., but the other, it's interesting.

A

Kind of bpf programs, which allows you to make decisions is,, as I mentioned,.

A

Like lsm bpf programs,.

A

And lsm hooks are very various.

A

You have like a bunch of lsm hooks triggered by I/o.

A

You have lsm hooks triggered by creating a new process.

A

Someone trying to log in as some user,.

A

There is really, there is,, they are really various and to all of them,.

A

You can write a bpf program in which you make a decision.

A

Whether you want the particular action to happen,.

A

Or you're going to deny it.

A

And well, bp, when you use bpf,.

A

You write bpf programs and then you need to write.

A

An userspace program, which is loading bpf programs into the kernel.

A

and loading of those bpf programs, is done by bpf system call.

A

So yeah,, usually when you, your project consists of two parts.

A

and so, you know, one is set of bpf programs,.

A

Which are usually compared to elf objects,.

A

And then you have userspace part,, which is loading them into the kernel.

A

And bpf has a concept of bpf maps.. They are data structures in which bpf programs can store the data.

A

and those bpf maps can be also shared to the userspace.

A

And there is a directory, sys/fs/bpf,.

A

In which you can pin those maps and.

A

Lets userspace programs to interact with them.

A

And well,, I will start with showing how.

A

Bpf programs look in pure c before going through the rust examples.

A

So I will most likely have to increase the font, right?

A

A

A

so, yeah,, the example in c is here.

A

So we have a very simple bpf program,.

A

(Inaudible) yes, like that.

A

(inaudible) okay., so it's a very simple bpf program.

B

Why are these cards, (inaudible) popping up?.

B

No, it's,, it should be turned off.

B

Okay,, I hope it will be going right. Now,.

B

A

So, there in kernel, there is a function called "clone,".

A

Which is called every time we are creating a new process., so basically, when we are doing fork.

A

In the userspace in the kernel, we end up with the clone function being called.

A

And we have here f entry type of bpf program,, which is tracing some function when we are entering it,.

A

So, at the moment of call.

A

So this bpf program is attaching to the clone,.

A

Kernel clone function., so it's getting triggered every time when we start.

A

Scheduling some new process.

A

and in this program, I'm only getting the pid of the process,.

A

Scheduling the new one, writing in,. Writing that pid in a bpf map.

A

And logging, the fact that something like that happens.

A

By bpf print k.- and this structure here is a bpf map in which I'm storing pids.

A

And here I'm making an action of saving the.

A

Particular pid into the bpf map.

A

A

Here I'm getting the name of the process.

A

So comm is like a name of the process in the kernel,.

A

And see right, here,, it's just logging.

A

and there is a userspace part,.

A

Which is loading the bpf program,.

A

Libbpf,, so the library in c,, which in which you are writing both bpf programs and loaders.

A

Has a concept of skeletons.? They are auto-generated,, so that schedule is auto-generated.

A

You instantiate the skeleton,, attach the bpf program,.

A

And then, yeah.

A

to keep it, to keep the program alive, a bpf program, alive,.

A

You have two ways.. You can either keep the userspace program. Alive,.

A

And as long as the userspace program is alive,,.

A

And we don't catch any signal. Here, the bpf program is alive., but if we want to keep the bpf program alive,.

A

Even after our loader.

A

Is not alive anymore,, we need to ping the program.

A

In the bpf file system., yeah, here,, I don't have an example for that.

A

In this example,, the bpf program is dying immediately when the userspace components is,.

A

Is gone. so I'll run,? Let me run it.

A

(Inaudible) yes, yes, yes.. I will end here as well.

A

like,. This is fine.

A

Okay. so yeah,. I have three examples already here,, but I will go to the one in c.

A

I will compile it.

A

Oh yeah,, it's already compiled and yeah, just to show it,.

B

I will remove the output.

B

And compile it again.

A

And important thing is that to do anything with bpf programs, we,, you have to be root.

A

So we need to execute.

A

The fork-trace, because that's the name of our program, is "roots.".

A

Successfully started, and now in the other terminal,.

A

We can use the tool called "bpf tour".

A

To check whether the program is really running,.

A

And we see it here. we have a tracing kind of bpf program.

A

Whose name is kernel clone.

A

And we can check the logs of bpf programs.

A

By doing bpftour/prog/trace.

A

And here we should see our output, that's yeah,.

A

Some process is forking.

A

So we have bash forking,, we have chrome, forking, (inaudible),.

B

The bash proxy, and, uh,.

A

Other processes doing forks.

A

And we are also saving dpi this in the bpf map.

A

So we can look, look up for bpf maps.

A

We have only one bpf map in the system,, which is called "processes,", so it's ours,.

A

And we can dump them up.

A

By name, processes,.

A

And yeah, we have pids.

A

I was in the program,, I was assigning value zero.. I was interested only in historic, historic pids, as the key.

B

So here it is, the proof that it works.

A

And yeah,, the presentation is mostly about rust, supports.

A

I showed how it works in c, as the reference point,.

A

But there are two ways and two separate libraries in rust,.

A

Which deal with bpf. the first one,, the more popular one, and in fact, official,, because like, affiliated with libbpf is literally like.

A

The bpf slushie, bpf-rs on github.

A

So,, it has pretty much the full coverage of bpf features.

A

And bpf programs,, but the thing is that.

A

When using libbpf-rs.

A

On the loader, userspace part is written in rust.

A

Bpf programs themselves are still written in c.

A

So when you are using libbpf-rs,, you'll maintain both c part of code for doing stuff in the kernel and the rust part for loading and for managing..

A

But there is also more experimental library, which allows you to do both things in rust,.

A

Which is called aya., it's very cool,, it's very promising,, but not every type of bpf program is supported. Yet.

A

And yeah,, it still requires a nightly rust.

A

And I'll start with showing example of libbpf-rs, where,.

A

So in case of libbpf-rs,.

A

The bpf program is still the same in c,.

A

It's even assembling here in this example,.

A

But we have a main rs module.

A

Which is using libbpf-rs to load the program.

A

Whoa, whoa, whoa,, whoa, whoa.

A

What happened? Here? okay, now it works. all right,, so yeah,.

A

So libbpf-rs automatically generates a skeleton.

A

From the c source.

A

so that fork trace skel builder is a structure,.

A

Which was automatically generated from that c code.

A

So yeah, you open the skeleton,, you load it, and then you have, that skeleton is providing commutable references.

A

To all the bpf programs in the particular module.

A

Here, we have only one., so we are just picking the kernel clone from procs mode.

A

We attached that program.

A

We attached the link and yeah, after that line, the program is loaded and it's going to be kept alive as long as the,.

A

As the userspace process is going, on.

A

And yeah, here is just code for like sending signals and like showing dots as long as the program is running.

A

So, yeah,: let's check this out.

A

I will go through the upper directory,.

A

And it's libbpf-rs/example.

A

here we have it., I will show, how is it building.

A

And there is a custom, like, cargo extension,.

A

Which we are going to use with libbpf-rs.

A

So we'll do cargo libbpf., so there is a libbpf cargo module.

A

And by doing cargo libbpf make,, we are firstly building that skeleton.

A

From the c code of bpf program.

A

And then we are building that main rs program.

A

So it's doing invites to steps, compiling bpf, objects, and like,, generating that skeleton.

A

and now we are actually using like the main cargo.

A

For compiling everything.

A

Will take some time.

A

All right,, so it's here,, and now we should have the ready to go binary,.

A

Which we can spawn and.

B

Target/Debug/Fork-Trace.

B

That should be it. yes, and it's it.

A

A

Let's check out what bpf programs are loading.

A

And yeah, we have that tracing program.

A

And we have also maps and we can bump them up again., and here we have bounce-off pids.

A

It's a bit shorter because I made it faster right: now.

A

A

Yeah,, we can also check the locks., so bpf prog trays.

B

Yeah., so we have like a bunch of forking processes.

A

And some new ones which just came up.

A

So yeah, pretty much the same experience,.

A

But the loader is in rust., but now I will show the example of aya,.

A

Which,, in which program, the actual bpf program is different.

B

B

A

So here I have a bunch of sub directories,.

A

The project tree here under aya directory is auto-generated;. They have a template which.

A

Creates those free sub directories., I didn't use the example common though..

A

It's like,, just a boiler plate,. If you want to have some common library for something.

A

But the two most important sub directories are aya/example,, which is our userspace program,.

A

Loading, the bpf program and here,, the actual bpf program,, which is getting clouded.

A

And the source code of the bpf program,.

A

As you can, see, is in rust., so it's doing the similar thing as the c code.

A

You've seen before, but yeah.

A

We have a definition of map here.

A

To which we are writing pids., we have the,.

A

The common practice in aya is that you write the main program, just like.

A

This,, that's one mesh, unsafe, (inaudible),.

A

And then you have separate, unsafe function where you actually do stuff.

A

Because bpf programs, there'll be always an unsafe context., so yeah, it's because of that.

A

Yeah, in that bpf programming,, we are getting the pid,.

A

We are writing a log and then,.

A

And then inserting the pid into a bpf map.

A

Yeah, we need to have, in case of bpf programs within aya, also need a separate panic, handler,.

A

Which you always assume that it's unreachable,.

A

And the loader part looks like.

B

It's loading slow.

B

A

All right. and yeah, the cool thing about aya. That's a feature that just came,.

A

I think, like three days ago, is that it has.

A

Like the aya log sub library,, which is showing here the logs of bpf programs in your loader.

A

and it's using underneath the perf event array, but, yeah,.

A

It's doing, it automatically for you.

A

It's,, it provides simple logging,, which we will see later., and here is the initialization of it.

A

Here we will load the,, um,, bpf, program.

A

And here we are getting the mutable reference.

A

To our particular program, we are interested in,.

A

Which is kernel clone.?

A

We then attach it to the kernel, clone function and here,. Here, it's just like, yeah, signal handling and showing dots that it's running.

A

So yeah,, let's check this out,.

A

B

Well,, similar to the other examples I showed,.

B

But yeah, it's all rust.

A

So yeah,, let's remove target.

A

And here we have two comments.

A

So we have an xtask, xtask to build the bpf program, first.

A

So let's do that.

C

A

A

A

It's taking some time.

A

and yeah, as I mentioned, it's using the internal llvm insights, rust, compiler.

A

And yeah, now we can build the loader, which is, you know,.

B

Just with cargo.

A

And as you can see, the userspace part here, is, yeah.

B

As in chronos,, it's kind of like tokyo.

A

Okay, we have it right now., so at the root we are spawning. The loader,.

A

It's in debug/aya-example.

A

And then it's an argument: we provide the bpf program.

A

To load,, which is under.

A

Target/Bpfel-Unknown-None/Debug/Aya-Example.

A

A

So it's running,, it's showing some logs here.

A

And yeah,, let's check the content of the bpf map. Then., the situation should be pretty much the same,, so we are going to have a hash map with processes.

A

And we have two new bpf maps., that's internal stuff handled by aya, to show you the logs here.

A

In the output of userspace program., so these two come from internal aya, sub library.

A

And so let's dump processes.

A

Here,, the output is a bit different because I'm, yeah,, it's still in hex,, it's in hex format,, but this would be pretty much like the same pids,.

A

Similar pids to which,.

A

Which we are seeing in the bpf programs and in c.

A

And c, of course, values zero,, because we always assign zero here..

A

We just assign.

A

A

funny thing, you know, aya, is still super experimental,.

A

And the reason why the log message.

A

Is only some process, forked.

A

Is that when I tried to log the actual pid,, the program was crashing.

A

so yeah,? That's how experimental aya still is..

A

There are definitely some bugs, but yeah.

A

At least I managed to save pids in the bpf map.

A

So that part worked, but logging, it was not possible.

B

A

And, okay,, that was it when it comes to.

A

Libraries in rust for handling and writing bpf programs.

A

Now I wanted to just tell a bit about the project.

A

I'm working recently., so I'm working also on quite fresh and experimental.

A

Project called lockc,, which is trying to use a ebpf lsm type of programs.

A

To make containers more secure.

A

There is a phrase about containers, containers, do not contain.

A

and the phrase means that, contrary to popular belief,.

A

Containers are not as secure as vms, and they expose a lot of stuff from the host.

A

And yeah, yeah, you can question whether it's necessary or not, but, well,. There are two points which are more important,.

A

Like most frightening about containers, today.

A

Some is file, systems are not namespaced.

A

So, okay, like when you create a container,.

A

You have some root fs image of some other linux, distribution, cool.

A

But some contents of slush/sys, and some contents of slush/proc is the same.

A

And to the degree that you,, if you go into sys/btrfs, for example,.

A

Inside container, when you are running,, when you are using btrfs as the fire system on your host,.

A

Then you are able to look up for metadata.

A

Of your like host partition., when you are in the privileged container,, you can normally use 'add disk'.

A

You can see devices,, you can see proc/irq and stuff like that.

A

And yeah, in my opinion,, it's not really necessary..

A

I would rather like the most of those sub-directories.

A

To be hidden from the inside container., so that's the one point which lockc is going to tackle.

A

The second point is that docker by default,.

A

Without any plug-ins, allows you to mount everything.

A

Inside container.- and it includes also your slash directory from the host and docker socket,, and if you mount those inside the container,, then you have pretty much passwordless pseudo.

A

And now combine that with the fact that.

A

Usually in the most docker installations,, you have a docker group and you add regular users.

A

To the docker group,, so they can run containers..

A

So, even though someone is a regular user.

A

With pseudo, with passwords,, he has granted for free password to the pseudo,, because he can just type dock and run a (inaudible),.

A

Or mount the docker sock.

A

yeah,, it's possible.. It's also possible to do chroot inside containers.

A

So yeah, the goal of lockc will be providing some kind of policies.

A

To prevent regular users from doing such actions.

A

I, yeah,, the goal of this talk is not to go too deep into the project. Itself.

A

you're,, if you're free, to, I'll, be happy. If you check out this project,.

A

But yeah, for now it's using libbpf-rs, because aya is lacking a support for lsm type of programs.

A

But as soon as aya gets that support,, I'm trying to work on that right, now,.

A

I will be more than happy to rewrite everything in rust.

A

And keep everything in rust and not bother with c.

A

So, yeah,, that was all for me,. This talk.

A

Thanks for listening., I hoped you liked it, and yeah..

A

Do you have any questions? (applause).

B

B

[student] yeah, the (inaudible, cuts off).

C

How do ebpf programs written in rust and c compare,.

C

Compare in size, for example, when you compiled that bpf program with aya,, I saw it, it compiled,, like the (inaudible) of rust.

C

And so on,, so I guess that's statically included.! So how big are these things, and how does it compare between rust and c?.

A

Rust form is, uh,, bigger,, yes.

A

It takes longer to compile also and.

A

It statically links some parts of llvm.

A

So some parts of llvm called a core library.

A

So yes, the rust bpf program is bigger.

D

Thank you for the presentation., so in the aya example that you showed,.

D

You mentioned that the rust code had to be explicitly marked as unsafe..

D

Is it because it links against libbpf or what is,.

D

What is the requirement that aya has that makes rust code unsafe?.

A

So I think it's that bpf programs gets the arguments.

A

From the kernel., so when we go to our program here, that context,.

A

You get that from the kernel and that context contains.

A

A lot of pointers,, which point to the memory,.

A

Which kernel allocated.? It's not rust, run time., so yeah,, that's because of dealing with pointers which we receive from the kernel,, and we have no idea, like,, which size are these, and yeah.

A

So that's because of, yeah, we didn't handle that memory allocation, rsl, secure.

E

So you mentioned that ebpf programs are written in a byte code, and I also noticed that they're loaded into elf files.

E

and I was wondering if you'd go into detail about what the byte code is and why it's in an elf file.

A

(Inaudible) knowledge about that, about such a low-level part of it, so, yeah,.

A

I'm worried I will be not able to provide any, like,, meaningful answers: here, but, yeah., usually the output of like compilation of bpf program.

A

By llvms, like the old file,.

A

Which has (inaudible),.

A

Which, 'cause, like the program itself in a separate section,.

A

Maps in separate sections.

A

and then it's getting loaded by libbpf into the kernel,.

A

And libbpf is also aware of, like the structure of it.

A

I'm sorry, I'm not able to provide any more detailed answer than that.

E

Well, neither could I, so,.

B

(Laughter) there's something to do with security,. I was remembering that you don't have everything allowed in ebpf,, because it's a insecure, context.

B

So the byte code doesn't allow everything.

A

That c would allow you to deal with in that context,.

A

So you can't break things so much in kernel., yeah. and ebpf, inside the kernel, there's also a verifier.

A

So when the byte code is loaded into the kernel,.

A

It's not executed immediately,, but first, verifier is checking whether.

A

You are, for example, not reaching the bounds of memories.

A

Kernel allocated for you., so it's firstly,, isolating, analyzing, the whole byte code., and when the verifier says, okay, that byte code is safe,, I'm okay,, I can execute it without leaking,.

A

Leaking memory,, then it's executed.

F

I didn't know about bpf before, and I just wanted to ask,. I don't know if it's even right,.

F

In the context, that,, is it constrained to any kernel, version, or like, recent kernel, versions?.

A

It depends on the type of program.

A

in general, ebpf. As we know, it, started being available around like four, one something.

A

I would say, like the oldest kernel,, which you can run to support the oldest type of network programs, for example, be able to use. Cilium is like,.

A

I would say, 4.18., you should,. It rather shouldn't go older than that., but then the more types of bpf programs started coming,.

A

They were coming in newer versions of the kernel,.

A

And, for example, the lsm type of program.

A

So the one which I'm using in lockc came in kernel point points,, uh,, 5.7.,.

A

So it differs on the type of program., the cool thing about bpf tool.

A

Is that it provides props,, so it can list you what,. It checks your kernel,.

A

And it provides a list of what kind of bpf programs are supported,. What kind of bpf maps are supported,.

A

And it also lists.

A

Functions,, which you can use in bpf, programs.

A

So, they're just touring around,, like checking what can be done with your kernel,.

A

But yeah, definitely the newer type of bpf program. Is,.

A

They are more concentrated about new versions, and if you want to use the newest stuff from like last year,,.

A

Then I would say, like five,, five points,.

A

Something 5.7 is necessary.

A

But if you are dealing with networking, like tracing packets,, redirecting them, doing some firewalling, ddos protection,.

A

4.18 should be fine.

G

(Inaudible) for the picture.

G

Okay, thank you., so you mentioned your personal project, lockc.

G

I also assume that you know apparmor from google.

G

So how is it different, any different from apparmor?.

A

So lockc is aware of each container, which is running.

A

And it will be, it's designed only for containers.

A

And so far, my idea is to have three different policy levers for containers,.

A

Restricted baseline and privileged,, where privileged will be really the kind of policy.

A

Which could be applied only by root.

A

and it will be discouraged to use it for anything else.

A

Which is not kubernetes infra,, so like, cni, plugins,.

A

Istio, service, mesh, or whatever,.

A

The baseline, that's restricted. That would be like the,.

A

The most encouraged to use and also able to be picked by regular users.

A

Of docker or kubernetes., the difference is that, well, in apparmor and sellinux for example, in regards to mounting files into the container,.

A

From my knowledge,, there is not really a way to differentiate what you can mount to each particular container..

A

You are rather globally labeling directories,.

A

Which you can, in general, mount to the container.

A

for example, in, for example, in case of selinux,, you apply a label to some particular directory which you want to mount,, but there is no differentiation or limitation to which particular container you can mount it..

A

So there is not so much awareness about particular.

A

Containers running. and, on the other hand, in lockc,, I'm maintaining the bpf map.

A

Containing all the containers running,.

A

Then all the, all the processes belonging on them..

A

So when I get the event coming from some particular process,.

A

I'm able to like look up for a concrete container.

A

And decide,, based on that, what to do.

A

And there is not so much awareness about particular containers in terms of apparmor,.

A

Selinux, or smack or another lsm system.

A

No more questions.

A

thank you,. Thank you. So much.

A