Cloud Native Computing Foundation Cloud Native Wasm Day North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Leveraging WebAssembly to Write Kubernetes Admission Policies - Rafael Fernández López, SUSE

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Leveraging WebAssembly to Write Kubernetes Admission Policies - Rafael Fernández López, SUSE

WebAssembly is an emerging technology that allows the creation of portable units of code that can be executed in a safe way, regardless of the architecture and operating system underneath. Many programming languages already support WebAssembly as a compilation target, and more are joining the ranks. This session will show how Kubewarden allows policy authors to write Kubernetes admission policies using the languages and tools of their choice. Policy authors can tap into existing tools, libraries, frameworks and best practices. This not only lowers the barrier to create policies, but also to review and maintain them. Policies are distributed via regular container registries, can be easily downloaded and tested outside of Kubernetes. This makes it easy to combine Kubewarden into DevOps processes. Finally, Kubernetes administrators can benefit from Kubewarden's integrations with consolidated Cloud Native projects to cover aspects such as distribution, testing and observability.

A

Hello: everyone, my name, is rafael, ferrande lopez and I'm a senior software engineer at suse and today, I'm here to talk to you about cube warden, which is a project that leverages webassembly to write, kubernetes and mission policies.

A

So what is keyboard and keyboard and is a kubernetes policy engine? It allows you to enforce security and compliance within your cluster and how do we do that? We do that by using webassembly binaries as policies, and so what these policies are going to do is they are going to receive a json object which resembles a request, we'll get some settings optional settings for their execution and based on that they are going to take a decision on whether they should accept the request, accept but mutating it or reject the request altogether.

A

So, if we think in isolation about this, then we are processing one request and we can either do the acceptance, rejection or mutation of that request. But how does keyword and intercept requests and provide responses with a decision, because this is targeting kubernetes? How does how does this work? How does this integrate with kubernetes and for that human test has an answer, and this is called dynamic and mission controllers, and by that we are able to register web hooks inside the kubernetes api server.

A

There are different ways, mutating and validating, and we can say on each of these workflow configurations in which kind of resources and api versions groups we are interested, and so, when we register these web hooks the current cps server when it gets a request from a client. In this case we have the cube, ctl cli tool being called, but they could be any kind of any kind of http request done to the apa server.

A

It will go ahead and we look for all the web hooks that are registered against this resource and operation, and so it will go through all the web hooks that are registered and it will produce an admission review for each one of them and then every webhook will have to answer to the kubernetes api server, give an answer to it and then the events, api server, will collect all these responses from all webhooks and it will produce the final response to the client.

A

And we know how to connect this with with kubernetes. We saw that we are going to use webassembly, and the question is: why do we want to use webassembly? So there are many reasons for that, and the first one is that it is sandboxed. So because keyboarding is a security focused project. We are very interested in that the interactions that happen between the guests. uh This is, in this case the policy and the host. That is the runtime running.

A

The policy are controlled and are only the ones that we want, and so the sandbox uh model of webassembly fits perfectly into this, but it's not only that it's also portable, and so when we we are able to build a policy, and we can build that in any kind of machine, any kind of architecture we don't need to cross compile.

A

We also don't need to have uh multi-art images like, for example, with container images, so it's portable and we can just copy and paste this policy in any kind of machine and operating system, and it will work as long as we have if this policy needs some washing interfaces also or some washi capability capabilities, then of course this is required, but the the binary itself is portable.

A

So this is super important for us and also it's a standalone, which means that I have this this binary and I don't need any kind of backing or or supporting library that I am dynamically linked linking to or the leap c. For example. I don't need any of that they are standalone. Binaries.

A

Also web assembly is extendable, so it allows us by custom by using custom sections in the binary itself to allow for reads embeddings. We will look more into that uh later, but the idea is that we we can. We can add more information, arbitrary information to the binary, and this information will go with the binary wherever it goes, and so this is very interesting um to keep important information with the policy that always flies with the policy.

A

So if we look at all these features, we kind of see that we are kind of describing like a universal binary right. uh So we have uh the way in which we we are going to um evaluate requests. We know what we are going to use to evaluate requests that is webassembly, but now we are going to look into. Why do another policy framework? What is the state of the art right now regarding policy frameworks?

A

So right now, and there is no right or wrong here. There are different ways to in to take decisions, and every company might take different decisions based on this, but this is important to take into account so right now. The policy frameworks that are widely used are based on the main specific languages, and so they they you have to learn uh about these domain specific languages. You need to learn about how to use them.

A

You need to learn about how um what they allow to you to what they allow you to do and what they don't allow you to do, and so this this has a cost like you need. uh You need to understand all these and then work on the uh and write uh code for this. For these specific languages, and also uh we have uh policy maintenance and distribution is not straightforward yet so uh we are right now you are deploying different uh deploy.

A

We are created, you are creating different deployments, different uh workloads running on different clusters, but policies is also something that you need to to to cover. Somehow you need to solve it in a new way in some way that you are already doing uh for other things, so this is also not a super straightforward solution in the in this sense.

A

So what happens? If, for example, we were thinking about what, if we create a policy framework that based on a universal binary, we could have kind of a universal policy framework based on that, and so this is what we are trying to aim for with your warden, because uh if you look at the left from the top to the bottom, we have open policy agent, which is uh which is um a policy engine as well. And you this.

A

You have to write uh policies in regal in the rigo language, and you can also try to use gatekeeper, which also you need to you to to write them, and you need to write policies with rego and also, for example, you could have you could use uh keyboard and sdk for writing policies in different languages that we support, like rust, go swift and will more more languages will come over time.

A

But the important bit here is that we are able to build uh these programs or these policies to the very same thing that is a web assembly binary, and so I am able to to build this. I have a local binary inside my machine in my inside my desktop or my laptop or whatever, and then the next problem is how I publish this, because I need to polish it somewhere right, and so there is a step before that. That is that we have to annotate the policy annotating.

A

The policy takes advantage of what we said before the custom sections, so we need uh we need to annotate with what resources this policy understands. uh A policy might be targeted in some um some api group of a specific api version, some specific kind of resource or a list of them.

A

It really depends, but it needs to to document what kind of resources it understands and since this information flies with the policy, the policy will always say what it is able to understand and what is disabled to take decisions upon and also we can by annotating. You can also add free from documentation to the policy. You can add also authors, for example, or the license or some other information. So now we have this file that we annotated and we have this policy.

A

That is also a file that is annotated and now the next step would be to publish it, and the keyword supports oca registries- and you are the good news- is that you are already using them, because you are deploying containers inside you insert kubernetes and you are pulling these container images from an ocr registry, and if you have any errors to your several of them, you will be able to publish these policies uh to the oci registry, and this is this works just out of the box, uh you just it's just an oci artifact inside of the oca registry, and they are next.

A

These policies are next to your container images. So you don't need to do anything special about about that. You need to do anything special to support policies inside your ocr registries.

A

Okay, so we see now that the storage is clear, but what about the execution?

A

So we have the policy server. That is the component that is going to actually run our policies inside of policy, evaluators or runtimes wasn't runtimes, and so the policy server when it starts it reads its configuration and see what are the policies that it should be running, and so, let's assume that this policy server on this on its configuration has three policies. For example, the first one is written uh for open policy again.

A

The second one is for gatekeeper, both are written in rego and the third one is written in rust, for example, with with the keyboard and sdk. But we don't care about that, because this these are all binaries for us, and so the policy server reads this information. This configuration file pulls all these policies and it starts an evaluator for each one of them and creates an endpoint for each one of them.

A

So a policy server has an http server, listening with three endpoints in this case, and then every endpoint is going to forward the whatever it. It reads to the policy that is running, and so then what happens is uh that the the keyboard and controller will register the workhook with the right endpoints inside of the policy server, and so you will have.

A

You will start seeing that the kubernetes api server is forwarding requests or admission reviews to our policy server and then the policy server, depending on the endpoint that was uh approached by the current cps server, is going to forward that to the right policy and then the right policy will do the evaluation. Instead of there was some runtime.

A

So we have this way of of working. We see how to deploy that and now the question is: can we do a little better regarding the cloud native intersection, and the answer is yes right now we have tracing support uh inside of the policies as well. So this is like first class citizens for us, you you, you have tracing support on the policy server. That is like the main entry point, but then every policy inside of the policy server is able to trace as they want.

A

So you can also see if something is taking more time than it should. um What are a request, since this comes from with a uid from the api server, the uid goes through.

A

The voice goes through the policy server and then goes through goes back, goes down to the to the policy itself and then, if the policy trace uh traces it somehow it will, it will get back to the policy server and the policy server will lock all these trace events, and so you can just open telemetry with a given collector, send that somewhere else- and you can also use jaeger for inspecting this and these distressing events also on our run map.

A

There is something interesting uh because we want to provide prometheus metrics and so by by providing permission, materials materials. We are able to show whether a policy is low in a lot or if it's rejecting a lot, we can see these numbers. We can see how many policies are running inside the cluster we can in general. We can expose a lot of interesting metrics that are very, very um useful for um for a humanities operator and then regarding the supply chain attacks. uh We have we on the roadmap.

A

We we have plans for supporting or foreseeing how we can interact with the six store uh project uh regarding policy signing because again, since we are security, focused africa, very security focused, then we would like to sign policies because in the end, uh policies are just binaries right inside the east side of annosia registries. Of course you can have some of them, but the idea is that we also support signing policies and now, let's go for a short demo, because I have two small demos to show all right. So now we are going to run.

A

Policies on cube on kubernetes and the print that we are going to solve in this case is going to be uh in our organization. We have this cluster that is exposed to production, so we are and we we have end users approaching this cluster and we have ingresses resources over there, and so the thing is that we are using search manager and we are using also let's encrypt, and we have two different ways of uh of requesting certificates.

A

One is with let's encrypt a staging cluster esswear and the other is with let's encrypt production cluster issuer. So what we want to do in this case is to by enforcing the annotations, because on the ingress annotation for cell manager, you can say what cluster is, where you want an annotation on the ingress, and so what we want to do is to limit this.

A

What is safe, annotation, what this annotation can have with the safe annotations policy, and so, uh if we look at the cluster admission policy in this case, this is the kind resource that we can deploy on the cluster and so the keyboard and will reconcile this. We say where our module is in this case. This is coming from an oc registry.

A

What are the settings that it supports in this case constrained annotations, which means that this key on the resource that is going to evaluate has to be of this value, and then we are going to target a number of rules in this case. It's just one and it started in ingresses of the apa group networking here on this io on v1 version and the operation that we are going to to to look after is create, and this is not a mutating policy. This is just validating one. Okay.

A

So let's deploy this uh this policy on the cluster. Where I already deployed keyboard then- and now we are going to wait for it to be ready, uh because now the the controller is going to to do some, um some paperwork for us regarding kubernetes and then at the end, it's going to register the web hook and when the hoop is registered, then it's actually active and now it finished, and now let's look at an ingress that has a let's encrypt production cluster issuer. So this annotation is what we are expecting.

A

This is an ingress that we would like to to let to let pass or to let us through, because this is right. This is fine. This will get a valid uh certificate, and so we are going to try to create this one, and this was properly created.

A

Now, let's look at another one in this case, as you can see this, this has this encrypted staking cluster eastward, and this would result in an invalid certificate for many of our users, because this ca is not trusted by by browsers, unless you manually, trust it, and so we're going to try to create this one and as expected, we get this error coming from the policy, because the this cluster is where value is not matching what we expect, and so this is uh how we can run uh policies inside of a cluster.

A

But now, let's say that you have already invested time in having gatekeeper policies or open policy agent policies. You can actually reuse them with keyboard and you don't really need to change anything. You just need to build them. So what I'm going to do? What I did is to build uh to create an echo policy which is super simple.

A

It would just do one thing what it does is it accepts some settings and the settings basically say whether it should reject the request or not, and if it rejects the request. What is the message it will provide back uh because of the reaction, and it will not even evaluate the request itself. It will just do whatever the settings say that that's why it's called the echo one, and so I'm going to build it, and for that I need the opa build uh command that comes with the oppa cli.

A

I target wasn't, and I provide the entry point for for my policy, and so I build and extract the policy that wasn't filed and now what I'm going to do is uh to run that locally. I don't want to to annotate and publish that into an oc registry first.

A

I just want to to run that, and I want to see if it runs as I expect, and so I run that with kwctl, and you can see on the settings json that I say reject false, which means that it's going to be accepted, and this is as expected, a load.

A

And now, if I do the very same thing but with the different settings are in this case we get true and with a rejected message, then I'm going to get a load false with the message that I am expecting from from the settings and also this part coming from the from the policy itself.

A

And now, let's do the very same thing. uh Let's run this this again, but in verbosity mode. So we see some things coming from the policy evaluator, but the important bit here is this bit here that is coming directly from the policy. The policy is going to the tracing call from from rigo and rigo is calling to the policy evaluator and the policy evaluator is reporting this through tracing. So this is working out of the box and now let me go back okay.

A

So some shout outs. uh What some time we are using this project to to do the runtime of of policies. uh We are also using ypc for doing the communication between the has the guest and the host, and we are also using the oca distribution crate uh that we use to pull and push to an oci registry. So we can pour some pull oca artifacts to to the oc to the oca registry, and this is part of the k, raster project. uh All of them are super useful and really nice communities over there.

A

So yeah really big, shout outs to them and the also be shout out to you so join the community. We have the keyboard and channel on the official kubernetes slack. You can join and look for some policies on the keyboard and hub. We we need to to to write more policies, so we welcome all the policies that you want to contribute. You can also report any issues inside of our org.

A

We have a number of repos over there and also some documentation that we have uh you can you can learn more about the project over there and you can also contribute to the documentation if anything is missing or is not clear and enough enough in in oblique clear, and with that, I will be open for questions. So thank you for your time and I really hope you enjoy this event. So thank you.