Cloud Native Computing Foundation Cloud Native eBPF Day EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: L3AF: Complete Lifecycle Management of eBPF Programs - Santhosh Fernandes, Walmart

Description

L3AF: Complete Lifecycle Management of eBPF Programs - Santhosh Fernandes, Walmart

- In this session, we will talk about how we can launch and manage eBPF programs using a daemon. At Walmart, we have developed a control plane that manages eBPF programs across a number of hosts running this daemon. The daemon can monitor and chain these programs (execute them in a sequence) in a pre-defined configuration state.
It achieves this by reading host-specific configurations that consist of eBPF programs attributes (artifacts, arguments, sequence IDs, policies/rules, network interface) to perform CRUD operations, akin to a linked list. In addition, we would like to discuss how the daemon leverages cilium's eBPF library to read eBPF maps and populate metrics specific to eBPF programs.

L3AF has been open-sourced under Linux Foundation (https://github.com/l3af-project/l3afd). L3AF is developed and managed by an enthusiastic community that is actively working on adding new features to it. We would also like to talk about how L3AF aims to provide a fully integrated software ecosystem around eBPF to unleash its full potential for community adoption across platforms.

A

Hello: everyone, I'm santosh fernandez, principal engineer, walmart global tech, I'm passionate about systems, engineering and distributor systems and building network solutions. Today, I'm talking about the product called leaf, which stands for lightweight ebp application foundation, which provides complete life cycle management of ebp programs.

A

A challenge we faced when we decided to adopt ebpf in our environment. How do you manage and orchestrate multiple ebp programs for a node in a large scale environment?

A

How do we deploy or change the configuration on the fly for the thousands of nodes in our environment? We have thousands of nodes, different cloud, vendors and multiple data centers to be managed, so there was no readily available platform to do similar functionality.

A

We decided to build our own platform leave, which does the complete lifecycle management of ebpa programs, which provides us a ebp program as a service.

A

It will give you a simple apis to add remote reorder, the ebf programs on the fly. It allows you to change the configurations of the ebp programs on the fly. It's also provide you, the matrixes generated by these eb programs to publish chromatic sql model.

A

It also replaces proprietary applications and the hardware with the blazing ebpf code.

A

So there is also a community driven ebp package repository where different capabilities, ebp programs are developed and uploaded so that once it is built we can use anywhere. It's a functionality of build ones, deploy many times manage a configuration, ebp programs per node basis.

A

It's also a cloud and vendor agnostic. It has got automation of configuration, management.

A

So much more capabilities in a leaf platform so going forward. We can go a little more deep dive on it.

A

This slide depicts about the leap platform on the right hand, side. We can see a uvp package repository, we can see multiple ebpm programs has been stored. These programs has been developed by the leaf community or a third party vendor, which provides you to manage or upload in the repository.

A

This runs in the build once and deploy anywhere philosophy, so that, once you compile and upload in the repository, then it can be deployed anywhere as and when needed.

A

In the left hand side we can see a leave d and we can see the apis to manage ebpm programs. So the any request comes to the lefty. It verifies for the ebp program artifacts and the artifacts to be downloaded from the ebp package repository and then it will deploy on the node. With the given configurations which are passed by the api, we can leave daemon orchestrate multiple ebay programs on the node.

A

It provides you xtp and tc chaining for the ebp programs. This is a core functionality of the leaf daemon. It also provides you a health information and metrics generated by the managed eb programs.

A

It enables, with the secured web api to manage ebp program.

A

Leave. Demon is a cross platform. Today, it runs on the windows. 2 protein leaf leaf, supports xtp and tc program chaining. It attaches the root program of xtp and tc to the interface.

A

When it attaches to the interface, it also creates the root map root program map, so that next program, which comes into the in the chain, will update its program fd into the root map.

A

And there is a tail call which will actually does the context. Switching between the first program to the next program.

A

How does we decide the location of the ebp program in the chain, so the location will be provided by the user in the config api through config api, it has got a sequence id the number which decides the location in the chain, and this entire chain is built on the concept of linked list and dynamically. We can reorder the chain.

A

We can stop the ebp programs. We can attach the ebp program in the chain like a linked list.

A

All these things are achieved using seriums go ebpf library.

A

Leaf provides to update the configs of the ebb programs on the fly, so no restart is needed for any config changes for the managed, eb programs.

A

How does the sample config update looks like it comes with a tag? Name map underscore arguments, and it takes the list of key values. The key will be the map name and the value will be the value to be updated to that maps.

A

The picture shows that leave the updating, the config ppf maps.

A

Leaf provides the capability to publish metrics generated by managed ebp programs. The leaf will read those maps and convert those data into the promulgal output format. How does it knows which map to read which aggregator to use? So these parameters are passed to the leave d from the config api, so the sample config api looks like the list of monitor map to that bpa program with the name with a key and the aggregator will be defined.

A

We support following aggregators max rate, maximum value for the given sample window size, average average of values for the given sample window size scalar the counter value, as is the window size, and the polling interval, can be configured in the leave d dot cfg below. Here we have depicts the sample promoquel output format, which contains different tags like host map name and the network function name and the value of the metric.

A

Leaf provides set of apis to update the configurations on the node, also to retrieve the ebp programs configurations on the node we have enabled mtls for our web apis mtls stands for mutual tls mtls runs on the zero trust security framework.

A

No remote connections are trusted by default. We support secure download of ebb package over the tls. This shows the sample config payload.

A

We can see the host name name of the host the interface which, for which these ebp programs has to be attached, and the bpa programs are categorized right, http, ingress or dc ingress or dc egress programs.

A

So we can see here, sequence id, which says the position in the chain artifact the name of the file which contains the byte code, the which will be downloaded from the ebp package repository map name, the name of the map created by this ebp program, to store the program fd for the next program.

A

Then. Similarly, we need to focus on the the version of the program, so admin status says that the program has to be started or stopped enabled means it'll start the program. Disable means. If the program is running, it stop the program.

A

Let's look into the demo now.

A

So we have a demo setup which, where the leaf is running in the vagrant vm, so we can see right hand side. The vagrant vm will be having leaf daemon uber package repository and web server.

A

On the left hand, side is a host which will be used to send request to the leave determined.

A

So we are using the head tool to generate the load on the vagrant interface and we have a browser to monitor the grafana dashboard.

A

The left hand. Side of the terminal represents the host side of this. The commands which are typed are executed on the host the right hand, side terminals, actually, the console for the vagrant vm. The command which are executed are executed inside the vagrant vm.

A

Let's see the config.iml here we can see those host port representation forwarding ports from the vagrant to the host ports.

A

So here we need to see the leave the code directory from where the the leaf d has been code has been picked up and deployed inside the vagrant vm. Let's start the yeah, so.

A

The lead is running here with no configuration, no eb program running. Let's check the swagger api.

A

This gate api will retrieve all the ebay programs currently running. Let's check that, so there is no ebay. Programs are running.

A

Let's go back to our terminal and send some load to the vagrant interface and see how it responds.

A

It took less than a second.

A

To run the 200 request and 20 parallel connections so less than a second, it could complete the request. Now, let's deploy the connection limiting and rate limiting ebay program, which will actually limits the the number of requests it can process to the by the backend service, so it will automatically slow request processing, let's go to our vagrant and use this try. This update api select and I have created the payload in my my notepad.

A

Let's talk about that, so I've set the.

A

Config map, as a max connections.

A

Per m per second is two for these two ports. So let's update it.

A

So it's successful, let's go and check on our you know so yeah these are running. Now we can check our vpn.

A

So you can see this so many maps are created by this ebp programs and if you want to see the particular map which we had updated here,.

A

A

You can see the two values set now we will run the hey tool again and we can see there is it took much longer than the first statement.

A

Yeah, at the same time, we can go and bi to check the number of eba programs running on the node.

A

So you can see here the details of the ebay programs running on the road check, our grafana dashboards.

A

A

You can see that it's taking much longer time.

A

It is dropping connections in the huge.

A

A

Okay, it's finally completed it took around 80 seconds driven projects if you are interested to work at the leaf. Here are some links to start with.

A