Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: What If... Kube-Apiserver Could be Extended Via WebAssembly? - Flavio Castelli, SUSE

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

What If... Kube-Apiserver Could be Extended Via WebAssembly? - Flavio Castelli, SUSE

Dynamic Admission Controllers are currently the only way to extend the Kubernetes API Server to implement security and conformance policies. They work great, but they also have drawbacks. How would it be to have a flexible way to extend the built-in admission controllers that doesn’t resort on Webhooks? This talk will show a prototype that leverages WebAssembly as a way to enrich the Kubernetes API Server capabilities. What if this is just the beginning of extending Kubernetes core pieces with WebAssembly? Do you want to join us in this experiment?

A

Hi everybody I'm flabby castelli from suzy rancher, I'm here today to talk about this crazy idea about extending the kubernetes control plane using webassembly, so for the ones who are not familiar with webassembly webassembly is a binary instruction format. That means you can pick your favorite programming language, write some code and then compile this code into what is being called a webassembly module. So, instead of building a linux binary for the intel architecture, you are instead targeting the web assembly architecture.

A

So web assembly is already being supported by many programming languages and more are more of them are currently working towards. You know supporting webassembly, it's actually a great time for web assembly. There is a lot of momentum. The reason is because webassembly is a is a pretty interesting technology with many advantages. So let's take a closer look at these web assembly modules, so the first thing you will notice about them is that they are pretty small, so these are small, independent computational units that take a really little space, and this is, I think, particularly refreshing.

A

You know considering you know how much space it takes a linux container, for example these webassembly modules, in addition to being small, they're, also portable. That means I can write some code, compile that, let's say on my apple silicon machine, I have a webassembly module and then ship this small webassembly module into.

A

I don't know a linux machine running on the intel architecture and then just run it that's because there is a thin abstraction layer in between the web assembly module and the operating system, which is the web webassembly runtime, webassembly, runtimes, historically speaking, have been placed inside of the browser, because web assembly has been a technology created for web application development, but nowadays this is no longer true and, as a matter of fact, there are ways to run webassembly outside of a browser, and we will come back to that later.

A

So there is one interesting point here, given that web assembly was conceived as a technology for the web. There are some nice side effects. The first one is security.

A

So if you think about that, when you visit a web page with your browser and this web page, has a web assembly module inside of it, your web browser is going to download this binary blob from the internet and then it's going to run it on your local machine. Does it sound scary to you a bit?

A

uh Luckily, when drafting up the web assembly specification, a lot of efforts have been put into the security implication of of that, and thanks to all this uh thinking, there are certain types of attacks that are not doable with webassembly others, but are harder to be pulled off. And of course I mean we're talking about security.

A

So I'm not saying that webassembly is the silver bullet. It has its own issues, vulnerabilities that are being found, but this is, you know, definitely a step forward, given that when it was designed, security was a centerpiece of a proposal, and it still is on top of that, when you are executing web assembly on your machine, each webassembly module is placed inside of its own sandbox. That means that it cannot interact with other webassembly modules running on the same machine.

A

You cannot see them, it cannot exchange data with them and in the same way, it cannot access the whole system. So you cannot see processes running on the whole system. It cannot access the file system, the network, the devices you can think about uh the sandbox. This bit of you know when you put uh the sandbox that is being used by linux containers.

A

From a certain point of view, uh you can remove some of these isolation features, for example, but there are other things that you you can't even yet do with webassembly, because it's not even supported right now. So this is an interesting thing to keep in consideration.

A

So I think, because of all of that, there is this interest of running webassembly, not just inside of a web browser, but also as a regular standalone application and, as a matter of fact, there are already quite some people that are experimenting and building systems that allow you to run webassembly functions as services.

A

So this is a great use case for web assembly. Another great use case is to leverage it to build plugin systems. So you start by you know it's a regular plugin system. You have a program, you want to add extra feature capabilities to it and you do it through plugins external plugins, but instead of you know, writing some native plugin one for linux, one for windows, one for mac for all the different architectures.

A

You just write it once you compile it into webassembly, and then you just use this webassembly module. So this is exactly what we are going to be talking about. Today we are going to expand an existing program and the program for today is the kubernetes control plane.

A

If you think about that, the kubernetes control plane has already some extension points. You can bring your own scheduler or what we're going to be more interested about today, you can bring our your own admission controller, so kubernetes, already ships with some admission controllers that are built into its own binaries, think about pot security limits quotas, all of that, but they cannot satisfy all the different use cases that kubernetes operators face on a daily basis.

A

Because of that, there are dynamic admission controllers that work in a really simple way, so the api server is receiving requests coming from users or our application, and then is going to send these requests to be evaluated to external web book servers. These webwork servers are usually running inside a very same kubernetes cluster.

A

They just receive this information about something that is about to happen, and then they and then they answer back with with a response which could be yes, accept the creation of this service or no reject the creation of this pod, because we do not allow privileged parts or yes, let's create this uh this part, but let's also add these labels because we have to be compliant with with these internal rules of our company.

A

So dynamic admission controller are a really well established mechanism of kubernetes and it is the foundation of many open source projects such as open policy, agent, gatekeeper, kyverno and, last but not least, cube warden, which is an open source project. I contribute to a project that we submitted for inclusion to cncf.

A

What differentiates keyboard and from the other projects I mentioned is the way policies are written and distributed.

A

Keyboard and policies are written with regular programming languages or, if you want, you can write them using rego, which is the query language created by open policy agent.

A

The difference is, these policies are then compiled into webassembly and they are distributed using container registries and then our dynamic admission controller provides a web book server, which is capable of pulling down these policies and using these policies to evaluate incoming requests.

A

So when I created the keyboard and project, I also thought about you know. This is just the first step. The next step would be, to kind of you know, eliminate this discrepancy, this differences between dynamic admission controllers and the ones bundled into into the kubernetes api server. So the idea I had was, would it be? Wouldn't it be great, you know to be able to extend the api server so that I can write my own custom policies, my own custom rules, but instead of you know leveraging an external web book server. Why don't?

A

I run them straight into into the kubernetes api server? So that was the idea, and you know you might be wondering why should I be doing that? I mean, after all, dynamic admission controllers have been out since a long time. They work great uh true true, but there are a couple of things we could gain by going in this direction. You know by further thinking about this idea, so the first thing is uh predict being more predictable.

A

You know so right now you have api server, which receives a request and then forwards this information to a web book server and then waits for an answer which could be accept, reject mutate or it could be nothing because there was a timeout because of a network leach or because I don't know the web book server is overloaded or it could be an http status code because there is something wrong going on with the web book server. So what to do next? This is where the failure policy setting that you can see on this slide.

A

This is what determines the next action for the api server, so you can pretend nothing bad happened, so you just ignore the failures and you just let this request go through, uh accept it, and maybe you got something you know. Okay or maybe you you got a workload instead of your cluster, that is dangerous and by the time you notice that it might be too late.

A

On the other end, you can be more strict and you can say I'm going to fail every time, and this is what the fail setting of where would would be doing, but in in in setting failure policy in this way your risk of blocking everything, because if a web book server is down or unresponsive whatever, then a lot of requests will fail, even the ones that are supposed to be accepted.

A

But uh there is no clear answer. Basically, you know there is no there is. uh There is no clear path towards you know, ignore or fail. It is something that you have to think about carefully. But what, if you you? You don't you wouldn't have to worry about that, and this is actually the case when you run all these rules or this policy straight into the api server, because you suddenly get rid of all kind of of a bunch of failure scenarios.

A

Another interesting aspect is- and this is kind of personal- I don't know how you feel about that. But I have this uh impression that a lot of cpu cycles and memories inside of kubernetes clusters are not being used to actually run user workloads but are instead taken by infrastructure code. So, let's think, for example, about the keyboard and stack, we have a webhook server, which is there waiting for requests to be evaluated. There could be a lot of requests if there is a lot of activity going on in the cluster or there could be nothing.

A

You know- and this is you know, cpu memory being you know, kind of wasted, just waiting for something to happen, and the same happens with the controller, which is there reconciling custom resources. And if there is a lot of changes in terms of policies being added removed, updated, then you know a lot of activity is going on, but otherwise you know it's again. Something which sits idle- and I feel like we came to you- know, accept this kind of situation, but there are scenarios where you know it feels better.

A

It feels good to not have all these overloads. So what if we could get rid of the web book server and the controller, because we have a way to run policies inside of the api server and basically, we have kind of a function as a service platform inside of api server. Just to evaluate all these policies.

A

I think if we go down that road, we would have you know more resources for our end users, and that would be great inside of constrained environments like edge clusters, where the hardware cannot just scale up and down in an easy way, because you have to actually send someone to add new boxes into these faraway locations.

A

So last but not least, there is one final reason why we should maybe keep toying with this idea, which is it sounds fun. Doesn't it at least it sounded fun to me and that's why I went ahead and I did uh a poc of it. So big disclaimer, you know, I know this topic is going to be a bit controversial. There are going to be people who are going to you know like this idea. Other people are going to be against that and it's it's totally fine.

A

So there are good chances that you know this idea will never be carried forward because everybody is against that, and so all this code is going to be dropped. I don't want to maintain a fork of kubernetes just to have that, and so what does it mean? It means that this poc I've been working on.

A

It had to be done in a short amount of time, and to do that, I try to reuse as much as possible what I know about kubernetes, what I know about this problem space and what I have already at my hand.

A

That means I reused keyboard and technology pieces of a stack like all the policies we already have with our web assembly, so I just want to run them, and but you know, if we want to move forward with that, that doesn't mean we have to stick with the keyboard and stack not at all all right. Having said that, uh this proof of comp set, I created, is basically introducing a new feature gate of kubernetes, which has an understanding of you, know, web assembly policies and uh the kind of contract that uh keyboard and policies offer.

A

So without uh further ado, let's, uh let's see this uh proof of concept in action. So here I have a k3s node, uh which is running. uh You know the vanilla stack of k3s. There is no keyboarding involved over there.

A

This is, however, a k3s binary, which has been patched with the code of this poc, so the feature gate I was talking about is turned on and this feature gate has a configuration file. So let's take a look at that, so here you can see that we have one policy called privileged. The policy is provided by this webassembly module, which is downloaded from from a container registry.

A

This is a vanilla keyboard and policy, the very same one you would use for the keyboard and running in as a dynamic admission controller. This policy is interested about. Pods uh is interested about creation, update operation against pots, and this policy is enforced only inside of a default namespace.

A

Next to it, we have another policy, called trusted, repos again taken from a container registry. Also, this policy is interested about operation related with pods, create update, and this policy is enforced inside of the default namespace as well. This policy can be tuned by operators, so it has some settings and this policy is going to prevent the creation of pods that have at least one container image coming from the docker up. I think this is a pretty common use case as a policy that I see in any organization.

A

So let's try to create a privileged pod. Let's try to create it inside of the default namespace and, as you can see, it fails, it is rejected and we get a message coming back from the api server, with the reason why this uh why this operation has been rejected. This is the same user experience you get when you're dealing with a dynamic admission controller.

A

Now, let's try to create a container sorry, a pod which is using a container alpine coming from the docker up again, we try to create that inside of a default namespace and again, as expected, we get a rejection because the policy is preventing us from doing this operation now to show you that policies are enforced only inside of the default namespace. Let's try to create a privilege pod again, but this time inside of the test namespace.

A

As you can see this time, everything worked as expected because the policies are enforced only inside of a default namespace. Now this poc code, it does one extra thing. So, every time a request is is rejected, because there is a policy which is uh which is against the operation.

A

The code is going to emit a kubernetes event of a certain type, and now, instead of um you, know, writing cube cti get events whatsoever. I just created a cubectl plugin that does that.

A

So now, as you can see, I have a fancy table which tells me all the recent events about policies that have been blocking some some operation from happening. Now. There is one interesting thing over there, this uh cuborden plugin of cubectl that I wrote is not written using go like or any regular programming. Language like you would normally do, but is instead delivered as a web assembly module. So this is another poc. I don't want to talk about today, but we can talk about that in the q. A if you want to.

A

Basically, I think webassembly can be used also, instead of other places of kubernetes core ecosystem like, for example, writing plugins for cubectl. I'm going to give you a link to this poc at the end of a talk.

A

So now, let's talk talk a bit about the challenges that I've faced while working on this poc, so the biggest challenge has been actually to to be able to evaluate webassembly inside of the api server. That's because all the major web assembly runtimes are not written in go, they are written in ras they're, written in c. They do have bindings for go, and this is what I've been using as a first iteration, but as soon as I started to to use them.

A

I then realized that the final binary, so the api server, would not be a statically linked goal binary, but instead it would be a dynamically linked one and that was kind of an issue, because this is not how kubernetes binaries are being built. So in the first iteration I made some patches to the kubernetes build system. In addition to you know the actual code of the feature gate and that got me you know I got to the point where everything was working.

A

I had a container image which contained the api server binary, plus this shared library that provided the web assembly runtime, and that was working. I got it up and running. I said that minicube it was. uh It was all great, but I was not you know. I wasn't super happy about having this dynamically linked binary and have all these patches to the build system.

A

uh Next thing I tried given that, as I said before, I think this uh this is going to be great, for you know, edge environment, uh edge environment uh triggers to me k3, as I immediately think about that, and so I went ahead and I back ported the patches I did for uh kubernetes, I backported them into k3s, and then I realized that you know this share library thing was was a big problem for k3s, because k3s is not delivered as a container image is a single statically linked, go binary, and so I had to find a way to to get rid of this shared library, and so I, after some experiments, I was able to take the the go bindings of wasm time, uh the the webassembly right time from from the bytecode alliance and repeal them in a custom way with muzzle c and uh and then have a one-goal binary able to do that.

A

I wasn't super happy to be honest about you know having this custom wasn't tango built, but it worked. But then I stumbled over was zero. Y0 is a project from the from the istio folks. They basically are in the same. You know kind of problem uh that I had because they they are using web assembly to extend istio, to write rules for istio, but istio is written in go, and so I guess they have the same problems I described before.

A

uh They went uh pretty ahead into this problem and they came up with uh with with a web assembly runtime, which is written in pure gold. This is a fantastic project because you just include it you I you don't need to do any patching to the kubernetes build system or you know, maintain, fan uh custom builds of of wozentango.

A

You just have it working. You include it like any other kind of go library. So thanks a lot folks. This is an amazing project there. There is also another uh positive side effect of using w0, which is uh the pc. Now is no longer using seago and for the ones who are not familiar with that. Every time you see go you're encouraging some performance penalties. uh Whenever you know you invoke this external shared library code, so with y0, you don't even have that. So I'm super happy about that.

A

There are some things that are missing from the poc because, as I said before, I wanted to prove uh that this is an idea that makes sense that could be used that could be accomplished, but they didn't want to. You know, invest too much time on it, because maybe we're just going to be dropping that, because nobody agrees that this is a good idea. So because of that the poc is not able to do mutating policies, it would be trivial to add them.

A

Also, there are some features that are offered to the policies by keyboarding like access information from the kubernetes cluster. To do context, aware, policies or you know, add information from six or so you can. You know, write an admission controller which prevents image not signed by trusted party to be used inside of the cluster and many other things, but all of that is done on the host side and q borden. The policy evaluator of keyboarding is written in rust, so I couldn't just copy and paste the code of keyboard and into this poc.

A

This code, as b, has to be rewritten in go. It's not rocket science, it's just a matter of time, and I didn't have time to do that, but it's doable. If you want to also there is another important aspect so right now the api server is going to download the policies which are web assembly modules from container registries, and it's just going to execute them.

A

But this is a bit dangerous of course, so the proper solution would be to do something like keyboard and does, which is uh verify what is being pulled down from an ocr registry by using six store so that you end up running only the trusted policies that that you, that you trust- but uh I didn't have time to do that- it's good it would be easy as well last point performances.

A

So I didn't spend time doing you know performance test against this solution. There is also a big question mark about one zero, which is a uh not optimized for performances. It's written uh on their uh project. Page as well, so this is something that needs to be investigated more.

A

What should we do next? Well for me, the purpose of this token. This puc is first of all to to to kick start a conversation around the usage of web assembly inside of core pieces of kubernetes like the api server over there. What do you think about that is a good idea is a bad idea. Let's have a conversation about that. Also, I think there are other places inside of a world kubernetes ecosystem, where webassembly could be pretty useful. Think about the cube, ctl plugin.

A

I've shown to you before this is something that could be done inside of other projects as projects as well iopi. I sparked your interest in doing something like that, and we can also, of course, talk about that. There are other things that I think could be extended with webassembly.

A

So, having said that, this is all for today. These are a bunch of links that I'm going to share with you. So this is the source code of the poc I've shown to you. There is also a ready to consume k3s, build that you can download and run on your machine. There is also a link to crew wasm, which is this project that I I started that allows you to create cube, ctl plugins with webassembly, and so this is it thanks a lot for joining for attending and have a great day.