Cloud Native Computing Foundation KubeDay Japan 2022, 9 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Hello eBPF! Goodbye Sidecars? - Liz Rice, Chief Open Source Officer, Isovalent

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Hello eBPF! Goodbye Sidecars? - Liz Rice, Chief Open Source Officer, Isovalent

The Sidecar model is often used in Kubernetes deployments to add common tools like observability, service mesh and security into application pods.

eBPF is a kernel technology that lets us dynamically change the way that the kernel behaves - and it’s revolutionizing the way we instrument applications running under Kubernetes. Tools built with eBPF have the power to observe, secure and connect all the processes on a (virtual) machine, so there’s no need to inject a sidecar container into every pod.

Does eBPF mean we’ll be saying goodbye to sidecars? Attend this talk to find out!

A

Ohio, because I Mass.

A

So yes, my name's Liz rice uh I work at isovalent, which is the company that originally created the psyllium networking project that Katie and kohe already mentioned this morning and psyllium is proudly an incubating project and it's based on ebpf and I'm going to be talking about why ebpf is such a powerful platform for not just networking but also observability and security infrastructure, tooling, and why perhaps we no longer need to rely on the sidecar model going forward.

A

I'm, you may have seen me before, as the uh chair of the technical oversight committee at the cncf and I'm also involved in an organization called open UK, which encourages collaboration around open source in the UK and also internationally.

A

So, let's start start side cars. What do I mean when I'm talking about the sidecar model? Well, even before sidecars, when we wanted to build common infrastructure, tooling things like logging and tracing and security tools that we want to run in every application we used to have to either re-write or perhaps import code written in the same language as our application.

A

When we move to Containers, we no longer have to use the the same language as our programming language. We can put that common code into containers and run the same container in every pod, so every pod is instrumented in the same way.

A

That container is a sidecar container. We call this the sidecar model and it's been widely used for logging and tracing and security tooling, and service mesh.

A

That's sidecars! Now, let's explain ebpf.

A

The acronym stands for extended Barclay packet filter. But honestly you can forget that, because it does so much more than packet filtering that the acronym is really meaningless. Now, what eppf allows us to do is to customize the way the kernel behaves. We can make the kernel dynamically programmable.

A

It's a bit like how JavaScript allows us to dynamically change. The behavior of a web page.

A

Now, normally AS application developers, we write our code in user space and every time our user space applications want to do anything that touches Hardware, maybe reading and writing to files or allocating memory sending Network messages. All these things require some Hardware interaction and user space can't do this for itself user space apps have to ask the kernel for assistance in all these activities.

A

We don't normally think about this, because programming languages give us abstractions. So we're unaware that the kernel is being invoked.

A

Now, when we use ebpf, we can write a program that runs inside the kernel, that's triggered by an event and that event could be any function being triggered. It could be a trace point. It could be a network packet arriving at a certain point in the networking stack. Whenever the event happens, our custom ebpf program gets run and if we attach ebpf programs to different types of events, we can use that to observe what's happening in the kernel, perhaps even react to those events for security purposes or networking tooling.

A

In kubernetes, our applications are still running in user space, but they're inside containers inside pods there's only one kernel per machine or virtual machine, so one kernel per host and that kernel is shared across all of the processes. All of the containers that run on that host as before our applications. If they want to do anything interesting like sending and receiving networking messages, reading and writing to a file, the kernel is going to be involved.

A

So the colonel is aware, and perhaps involved in everything, that's happening on that host and if we attach evpf programs to the right events on the host, we can build tooling. That is aware of all of those applications running in all of our pods. On that host. We only have to instrument the host. We don't have to instrument every single pod, so I've talked about what ebpf is I've talked about.

A

What sidecars are, let's dig in a little bit to some of the advantages of ebpf over sidecars for a sidecar container to be aware of what's happening in a pod, it has to run inside the same namespaces, the same Linux namespaces as the application and to get it there inside the Pod. We have to have some yaml code normally that yaml is created automatically, perhaps by an admission controller or through CI CD, but suppose something goes wrong or there's some misconfiguration.

A

If the yaml doesn't get injected into that pod, the container won't be injected into that pod. There will be no instrumentation of that pod and then that pod is not visible. The tooling can't have any influence over that pod.

A

Ebpf doesn't need any app configuration changes at all because we're loading the ebpf program into the kernel and as soon as you attach an ebpf program to event an event it's going to be triggered. Regardless of whether the Pod was running already, you don't have to restart your pods they're, always going to be triggering those ebpf events. There's no need to reconfigure your pods, you don't even have to restart them.

A

Also ebpf is going to be aware of all the activity on that node, even if it's malicious activity, as well as the expected activity so tooling can see malicious processes running on the Node.

A

The cycle model can also be pretty wasteful of resources.

A

Every pod has to be configured to have sufficient memory and CPU not just for the application, but also for the sidecar container, and we need to have duplicate copies of state and configuration information inside every pod, because by Design pods are isolated from each other. You can only share information using shared files or network access network messages.

A

In eppf we have a concept called ebpf Maps. These are data structures that make it very efficient to share data between ebpf programs in the kernel and our user space, tooling agents, so lots of tools have already been written that take advantage of ebpf and its power. Let's look at a couple of examples from the cncf landscape.

A

The first I'll mention is Pixie. This uses ebpf to collect metrics and all sorts of different metrics from all the nodes in your cluster, brings them together and presents them in various graphical ways. This is just one example of the kind of information you can get with Pixie. It's a flame graph showing CPU usage across an entire cluster.

A

Another observability tool is the Hubble component of psyllium, which gives visibility into networking information. We can see individual packets and we can see the build up of a service map showing how traffic is Flowing between different kubernetes services, and we also get metrics information at both layer, 3, 4 and layer 7.

A

The application layer Hubble, is part of the psyllium project and psyllium also uses ebpf to provide efficient networking connectivity I mentioned that evpf can attach programs to events in the networking stack and we can use that to bypass parts of the stack to deliver packets much more efficiently within a kubernetes network and as well as providing connectivity.

A

Psyllium is also providing security using ebpf.

A

It gives transparent encryption, so we can encrypt traffic between nodes in the kernel and we can use ebpf to enforce Network policy by dropping Network packets that are out of policy and about a year ago we added some additional capabilities to psyllium so that it can act as a service mesh sidecarless service mesh.

A

Now, what's a service mesh, a service mesh provides connectivity between apps at a service level. It abstracts away the underlying Network and provides additional features like observability and security and traffic management, so the Innovation that psyllium brought to the world of service mesh is building it without sidecars, which we can do because of the power of ebpf.

A

Now all service meshes use a network proxy to handle and process traffic at the application layer, layer, 7. and other service meshes have used the sidecar model for that proxy. So there's a proxy injected into every application. Pod celine's implementation allows us to share a network proxy across multiple pods.

A

Avoiding the sidecar model helps to avoid the complexity and resource usage that has often been associated with configuring. The proxy in every single pod per Kelsey's quote about service mess.

A

Combining this with ebpf also gives us a much more efficient networking path and a shorter networking path means reduced networking latency in the sidecar model. Every single Network packet has to Traverse the networking stack multiple times to pass through that proxy, and if you have two pods communicating both having a proxy, then the networking path gets pretty convoluted because it has to pass through a proxy at both ends.

A

If we run a the proxy outside of the Pod, then we can create a much more direct networking path between two different pods, only passing through the proxy once and only passing through the proxy when it needs to because layer, 7 traffic is involved, and we know from psyllium beta testers that this ability to avoid the operational complexity and the improved latency is really attractive. There are many organizations for whom psychar based proxies. It's just been too high a price to pay for the functionality that they get.

A

Psyllium was the first to implement service mesh with outside cars, but now istio has also followed Suite with the ambient mesh approach. There are some differences in how psyllium and istio are approaching this, but they're both taking sidecars out of the equation. They both use Envoy proxy to handle the complexity of layer. 7.

A

sidecars have always been an unfortunate implementation. Detail.

A

Mesh features will move to the underlying infrastructure at that's from Matt Klein, who originally wrote the envoy proxy and with psyllium we've moved a significant part of that service, mesh functional functionality into the kernel itself, and we delegate responsibility to the envoy proxy and uses user space to handle layer, 7, complex processing.

A

This idea of running processing inside the kernel and delegating complex parts to user space has been done before we're doing it in networking in service mesh.

A

Similar pattern is used by suricata security tooling, which makes security decisions in user space based on packets sent to user Space by the kernel using nfq or if you plug in a hardware device in Linux, the kernel will call to a user mode helper, typically to help with the complexities of configuring that device when the kernel module gets loaded now in service mesh. There might still be reasons why you do want to co-locate your proxy inside the pod.

A

If you're running, complex custom wasn't filters in your Envoy proxy, you might want to isolate that proxy from other pods. So you could continue to do that using the sidecar model and I think we'll actually see service meshes, offering a range of options for how the proxy is isolated. How many instances of the proxy exist.

A

The last part of the service mesh puzzle is encryption.

A

Mutual authentication between application workloads is a really important feature for many organizations.

A

Let's have a look at how we're handling that in psyllium using ebpf. So I already mentioned that we support transparent encryption traffic, that flows between different nodes gets encrypted in the kernel, using either ipsec or wireguard, and this is has been really great for many organizations because all traffic is encrypted and it's very easy to manage, and you can also use psyllium Network policy to ensure that the traffic that flows between different Services is permitted, but we're now working on a next generation of encryption, Mutual encryption using application workload identities.

A

So you can use the identity management system of your choice, spiffy, cert manager, whatever else to manage the identities and the certificates that represent each individual application. Workload and psyllium will inject those certificates into the kernel and we'll use kernel level encryption for encrypting the traffic.

A

So we've solved a lot of the problems in service mesh in observability and security tooling. Using ebpf. Does this mean we will never see sidecars again, I'm sure we will still see some use of sidecars. First of all. Ebpf program is essentially kernel programming. So there's a pretty high barrier to writing. Ebpf programs. I. Don't think people will immediately rewrite all their infrastructure tooling using ebpf, and certainly the sidecar model will be a lot easier for many developers to experiment with the other reason to use.

A

Sidecars will be, if you don't have access to the underlying node in many managed. Kubernetes environment you'll need the cooperation of a cloud provider.

A

If you want to use ebpf based tooling, but the power and the efficiency of ebpf means I'm sure we're going to see much more infrastructure tooling built on that platform, and one of the real reasons for, for that will be the pull of users who demand improved performance, and we see proven performance improvements because of things like reduce Network latency, because we can run so efficiently in the kernel and because we can load those ebpf programs, dynamically.

A

So I think the sidecar model still has its place. We will continue to see it used in some environments. I believe ebpf has a huge role to play in the future of infrastructure, tooling, for networking, for observability and for security.

A

If you'd like to learn more about evpf I'll be giving away copies of a book called what is ebpf that I wrote we'll be doing that in the networking session this evening, so I hope, you'll come and say hello and we'll we'll get you a copy of that book. Thank you. Arigato gozaimas.