Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes-native Security with Starboard - Liz Rice & Daniel Pacak, Aqua Security

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes-native Security with Starboard - Liz Rice & Daniel Pacak, Aqua Security

Starboard is an open source project that gathers security information from various different tools into Kubernetes CRDs, so users can manage & access security reports through familiar Kubernetes interfaces, like kubectl or Octant. This talk uses plenty of demos to describe the motivations behind Starboard, and design challenges such as: - how reports can relate to different resources, from pods through to entire clusters - showing security information for running workloads - extensibility and plugins Learn how to use Starboard generate to actionable security information that's visible to the people who need it. See how to extend Starboard to integrate your favourite security tool, if it isn't already covered today. This “Advanced” talk won’t hold back from showing you the code! It assumes basic familiarity with security tools like vulnerability scanning, YAML checks & CIS benchmarks.

https://sched.co/ekE7

A

Hi there, my name's liz rice, I run the open source engineering team at aqua security, where we build tools to help enterprises secure their cloud-native deployments, and I'm here with daniel patzack.

B

Hi everyone I'm a software engineer at aqua, taking care of our open source projects, including starboard.

A

And today we want to share with you some background on what starboard is and uh describe some of the design decisions that we've made as we've been developing it and then talk a bit about the way we see starboard developing in the future.

A

So, first of all, the the motivation behind starboard meet our friend dave lopez. He uses kubernetes and to do so, he uses tools like cubecontrol and perhaps he uses a dashboard like octan or another kind of ide interface that uses the kubernetes api to access the cluster and manipulate resources within that cluster.

A

Now, if dave is also interested in security today, he has to learn to use a variety of different security tools. They all have different interfaces. They generate output in different formats. Maybe some of it is in html reports and some is in json files.

A

The idea behind starboard is to bring all these disparate tools into the world of kubernetes, so we've created the starboard cli tool that provides an interface, a familiar queue, control, plug-in interface. That dave can use to run security tools and it creates the reports in the form of kubernetes custom resources that dave can access using cubecontrol and the dashboard so over to daniel to show us the cli in action.

B

Right so I'm going to demonstrate how we can run a cube bench as a security tool via the starboard cli interface.

B

What I have here is a local kind cluster with four nodes, three worker nodes, and uh we also have to install starboard binary in here, I'm installing it from the crew index and we have to initialize starboard it's one time command. It is creating all the crd definitions and service accounts and configuration map, and with that we could trigger cube bench scanner.

B

Since we have four nodes, we are spawning four kubernetes jobs which are scanning and running benchmarks on each node. Once it's done, we could list the results. So, as you can see for each node, we do have all the checks run, and we can see immediately that there are some failures for those who are more familiar with user interfaces or graphical user interfaces.

B

We also developed a starboard plugin, and this plugin allows us to navigate to the list of nodes, select a particular node and we contributed a cis kubernetes benchmark which simply shows the same information, as you saw in the terminal.

A

So the cli allows our friend dave to run security tools manually through cubecontrol starboard. The next step is to automate this using an operator, so we've created a starboard operator at the moment. It just runs vulnerability scanning on workload resources, so the operator watches for new pods starting in the kubernetes cluster, and it runs a vulnerability scanning tool. We have a couple of options already implemented and we'll be talking about that later.

A

Having run the scan job, the operator writes the vulnerability report, custom resource so over to daniel, to show us the operator in action.

B

Yeah, so let's take wordpress as a sample application. uh What you can see is a deployment descriptor, but this time I'm going to use a octant apply, yaml file feature to create this deployment, and now, if we switch to the starboard operator nation is this is where we run. This is the namespace in which we run the operator itself and in the same namespace it spawn immediately a scan job.

B

It is annotated, so we could actually figure out which workload it is uh scanning um and it's related to the active replica side of the wordpress deployment.

B

If everything is fine, this job is automatically cleaned up and the operator will create a vulnerability report and associate it with this replica set.

B

So we could also go and see what is going on and what is the descriptor of this image? There is an init container. There is a wordpress container, that's actually the command that we are running and since octant is automatically refreshing, the ui it has been completed. Switching back to the default namespace wordpress- and here is where we make this vulnerability information available. There is a status card component and uh the plugin displays the stats. If you want to gear down drill down, we have the vulnerability tab and here is a list of vulnerability items.

B

The operator will also act on the rolling update event. So if we bump up the version of the deployment to see if the newer version has less critical vulnerabilities, we will see that in the same way, there is another scan job in the starboard operator namespace.

B

It is using a different set of labels, because now the active replica set for the application has changed, but if everything's goes well, you will see the report updated.

B

In here, for now it is still um scanning, and here we are, the latest version has only two critical vulnerabilities and um again we could drill down and see all the details here.

A

Thank you daniel, so now you've seen starboard in action. Let's talk a little bit about some of the design decisions that we've taken while building it and you've seen how security report information is created and is associated with particular resources and we're trying to generalize handling different types of security report associated with different types of resources, as you saw we're using labels on the security related resource to identify which kubernetes resorts it relates to.

A

So we can use label selectors to extract the right security information for a particular kubernetes resource, but we're also using an owner reference, and the good thing about this is that when the owning resource gets deleted, the associated security resource that owns it also gets deleted.

A

We simply don't have to worry about garbage collection, because kubernetes takes care of it for us deciding to use this owner. Reference approach actually settled a whole other design decision for us. We've been going back and forth on whether it would be useful to maintain historical security reports and by using this owner reference, we know we're going to be cleaning up security resources when the associated resource is deleted.

A

So we can't hold on to historical information, and actually that makes a lot of sense, because kubernetes isn't intended to hold a historical database of things that have happened in the cluster in the past. So there's no reason why it should hold on to security reports from the past either.

A

There are other options for that log, the information and and store it elsewhere outside of the cluster.

A

We also had to make a decision about the name we give to each of the security report resources.

A

Initially we sort of using a uuid because it would definitely be unique, but it's a random string, that's meaningless to humans. The alternative would be a deterministic name, but when we were still thinking about historical reports, we worried that it would be more complex.

A

We could. We could solve the problem, but to make those names unique uh would involve doing something like including a time stamp if we wanted to make it a human readable name.

A

So we initially led towards the simple solution which was just to use uuids and know that they would be unique once we've decided actually we're not going to store historical reports. There will only ever be one security related custom resource related to a particular resource.

A

The whole naming issue stopped being so complicated. We could use deterministic names because concatenating the resource type and its name gives us a unique id within the namespace and it's meaningful to humans, but it also had a useful implication in the implementation right. Daniel.

B

uh Yes, uh since we are using the controller, runtime library as part of the starboard operator codebase, which is using a pretty advanced kubernetes client with its own cache, this deterministic names solve the problem of duplicate vulnerability reports that we happen to create from time to time, because we were not leveraging this whole caching mechanism.

B

So the eventually the resource name like a deterministic name, reads better in the cli interface, but also improves the reliability and implementation of the operator.

A

When we first spoke with customers about starboard, something they brought up very quickly was role-based access control. Generally speaking, dave should only have access to the security information related to the resources that he has access to.

A

So putting those reports in the same name space as the resources can make our back configuration pretty straightforward, but, as you saw, starboard needs to run jobs and we have to decide what namespace to run those jobs in, and we decided to do that in a separate namespace for starboard. For a couple of reasons.

A

First of all, it means starboard doesn't need permission to run workloads in your application namespaces and that's better from the point of view of the principle of lease privileges. We want to give everything as limited permissions as possible, so snowball can only create these jobs in its own namespace.

A

The other advantage of this is that, when dave is looking at the applications running in his namespace, it's not cluttered up with the occasional uh scan jobs that starboard the starboard operator would be automatically creating.

A

Now for the cli, the scan jobs run in a starboard namespace and for the operator they run in whatever namespace you're running the operator in right.

B

Yes notice also that not every resource building in kubernetes is is scope to namespace. uh There are nodes and for the- and you saw in the demo that we run uh cubebench for each node and then we associate a report with a node. So in this case uh we also distinguish uh between namespace and cluster scoped custom security resources. So vulnerability report would be a nice burst report, whereas already mentioned cube bench report or cube counter report is a cluster scoped resource.

A

So we've seen that in general we're associating a security report with a resource and when it comes to running workloads, we actually had a few options to consider.

A

Remember that we're trying to make it easy for dave lopez to find out the security information about his workloads, his running applications.

A

Now, if he runs an unmanaged pod, it's pretty straightforward that the vulnerability report is associated with that unmanaged pod. But if we're talking about deployments, there could be multiple instances of each pod and if we were to associate vulnerability reports with those pods we'd have duplication.

A

Those reports can actually be pretty large, so it could turn into a practical storage issue to store vulnerability reports at this level. So we decided not to do that.

A

An alternative might be to think about associating the vulnerability report with the deployment. After all, that's the resource that dave is typically going to be manipulating, but there is a problem with this: there isn't always a single replica per deployment and if we're, if we have multiple replica sets, they may have different images in their pod specs. So we need different vulnerability reports for those images. Those different images that they refer to so the conclusion is, we need to hold vulnerability reports associated with replica sets when you come to think of it.

A

Unmanaged pods can also be replaced with a pod of the same name but a different image. So for that reason we include a label with a hash of the pod spec so that we can tell if the security report for a particular pod is out of date.

A

Even if we're storing these vulnerability reports per replica set from dave's point of view, he's probably interested in queering it related to his deployment.

A

So we've made it easy to traverse the hierarchy so that you can make a query at the deployment level and it looks up the vulnerability information from the associated replica set and over to daniel to show us the hierarchy.

B

All right so just to show you uh the current report that we have in our cluster. uh We scanned uh all the nodes with cubepench and, as you can see, we set the owner reference for this report to point to a kind uh worker, node, and but also you see that it's because of this great control trip plugin. We see the whole hierarchy of objects, it's even more interesting when we display it for the wordpress application.

B

Since we did a rolling update, we have two replica sets. One of them is active as you can see this ready status here, and we also created a two vulnerability reports which are linked back to the replica set and just to show you what we mean by traversing: the hierarchy of objects. So remember that, even though we don't have the report associated with the deployment object, we do display this information right because we can do programmatically, traverse the tree and display all the vulnerability information right in the user interface.

A

So we've talked about how we envisage starboard being extendable and being used to show security reports generated by multiple different tools. Now, let's talk a bit about how what you would need to do if you wanted to add support for a new security tool into starboard.

A

B

uh uh So here what you can see is uh like a quick explanation: how starboard schedules uh jobs and uh how you can contribute or security vendor can contribute with its own vulnerability scanner.

B

It's not like a plug-and-play experience yet, but in general we believe that we, we could support other tools out of the box, like we have this reconciliation loop, that is constantly watching uh deployments or pods that are created, and then starboard is taking care of scheduling a job, uh adding all these labels that we explained and eventually persisting the uh the report, and here what we expect from the vendor is to implement a simple interface to provide a pot template spec.

B

This is where, as a prerequisite, we expect that your vulnerability scanner is containerized, so it could give us its image and then also build up the command that is used for scanning in case of 3v, which is the default vulnerability. Scanner in starboard. The command is pretty simple: it's like a image reference and then the 3d output converter. This is a piece of code that is reading a controller pod and it's transforming the previous model to the starboard model right, which, in this case is a vulnerability report.

B

It has its own schema, but we can't know what is the model of the third party tool. So this is these two bits have to be implemented separately and if we move to the next slide, you will see a simple interface called vulnerability. Scanner with those two callbacks one is get bot template spec, that's the one where you specify.

B

What is your image reference and what is the actual command and all the args and the second method is a callback to parse the stream of logs coming from the pod.

B

If you intend to reuse all the custom security resources that we defined as part of a spar starboard, you could also swap, for example, polaris, which is used by default as a config audit tool.

B

So as long as you comply with this schema of config audit report, you could reuse a for example. Plugin. You don't have to build your own dashboard or something like that. Just fit the data with your tool and we will visualize it for you.

B

The same applies for other ides, for which we don't have plugins now, but since the crd is a common thing in kubernetes and since we added additional printer columns, even the default ui for displaying custom resources in lens allows you to quickly spot that in a given application, you have 91 critical vulnerabilities.

B

Of course we can wrap it into a plugin, but it's it's already useful and also you can see here all the labels and the naming conventions that we were talking about.

A

So today, starboard is extensible, it's possible to add support for new security tools. You do need to write some code where we would love to get to in the future. Is the ability to just add new security tools through configuration?

A

Imagine that it's really a simple case of saying or telling the operator I want to watch a certain type of resource and when there are changes in that resource, I want to call a particular security tool and it will generate custom resources of this particular type.

A

So adding in support for a new type of security tool would be a case of creating a new custom resource definition and adding in the the definition the configuration for that new tool. So that's the vision of where we want to get with plugability. Today it is a little bit more complicated. You do need to write a bit of code, but this is this is the ultimate goal, and we would also be very keen to hear feedback on the customer resource definitions that we've created so far.

A

We hope that they're flexible enough to plug in other alternative tools, but we're very open to hearing feedback last thing that we wanted to talk about in terms of the future of starboard is helping dave lopez. Ask the question: what are the most security most important security issues in my cluster or what are the most important security issues in the particular namespace I care about.

A

We want to get to a point where we can summarize the most important issues from across these different types of security report and use starboard, to make it very easy for dave to find out what security issues he really needs to worry about.

A

So if that sounds interesting to you, you can download and run starboard from github. It's also available on the artifact hub and the operator hub, and we would love for you to get involved. Do you come and check out the starboard repository on github? You can get in touch with us or reach out to us here at the conference. We'd love to hear from you. Thank you very much.

B