Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Threat Modelling: Securing Kubernetes Infrastructure & Deployments - Rowan Baker, ControlPlane

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Threat Modelling: Securing Kubernetes Infrastructure & Deployments - Rowan Baker, ControlPlane

Security teams are often the last to know about an installation of Kubernetes, and are frequently concerned by its adoption. They have every right to be: poorly architected clusters can easily become exposed to unexpected threats, compromised by hostile workloads, or impossible to maintain. It doesn’t have to be this way! This talk details mechanisms for architecting Kubernetes securely in regulated organisations, and shares lessons learnt threat modelling with the CNCF Financial User Group including: - How to use threat modelling to secure clusters and workloads - Real-world examples of Kubernetes deployments, and multi-tenant security architectures from financial services - Where to apply controls to layer defence in depth - Using compliance standards to satisfy security teams - How to integrate Kubernetes with a global SOC - Gotchas, common threats, and advanced mitigations

https://sched.co/Zeow

A

Hi, I'm ron welcome to my talk on threat modelling securing kubernetes infrastructure and deployment, I'm the head of security at control plane and we're a consultancy specializing in cloud-native security out of london. I'm also a contributor to the gke cis benchmark and cncf financial user group kubernetes detectors.

A

I need to start by acknowledging that a lot of the thought behind codified controls, automated testing and compliance and the role of attack trees presented here are those of john meadows who founded the cncf financial services user group. We were supposed to present together today, but unfortunately, couldn't owing to circumstances beyond our control.

A

In this session today we're going to introduce threat modelling and threat modeling kubernetes we're going to show how your threat model helps define your control sets subsequently we'll look at how tests associated with your threat model and controls, assist with sock integration and addressing compliance, culture shock and we'll finish with some gotchas and a tldr.

A

So, to start with a baseline, I'm going to very quickly describe the threat modelling process with the 5ws and how, in fact, if this is all you take away from the talk and act on with your own kubernetes implementations, that's a good thing.

A

So what is threat modeling? It seems to be used interchangeably as a noun and a verb.

A

Well, the definition of threat modeling will use, takes the form of the verb, and it's not just about defining the risks to the system, but about how they are addressed. So why should we do any threat modeling at all, well threat. Modeling can prevent you from finding out about security issues when it's too late, if done properly, establishing a threat.

A

Modelling program provides the opportunity for the right kind of cloud: kubernetes migration, where your developers drive from a capability and experience perspective and security compliance developers work in tandem to enable that to secure compliant fashion, using the threat model effectively. This means shifting security left in the development process and identifying the security requirements for the developer.

A

So you should really start threat modeling as early as possible. As soon as a shared understanding of an architecture starts to develop, you should be threat modelling.

A

This provides real benefits if you find a security issue early and are able to change the architecture to be more secure before anything has been implemented, you'll have saved a huge amount of rework and likely arrived at a more secure and usable implementation.

A

If security issues are picked up later in the process, you'll be trying to retrofit security into the architecture.

A

You may be successful, but the chances are you'll end up with a more complex and compromised solution, so threat modeling just doesn't stop with the first release once you've got an in-flight project with an existing threat model when the tickets and features are defined for each release, evaluate and update the threat model around those specific features and tickets.

A

So who should contribute to your your threat model? Basically, everybody each day told us something to contribute and their own unique view of the world. Now architects have a vision for the system.

A

If that system is already in use, then devops will know how that works in practice and they'll know about those little tricks and deviations. They have to get things working that may stray away from the the intended architecture.

A

Your security operations center, your pentest teams, your threat, intelligence teams are also useful people to have on board these people know about the tactics, techniques and processes that attackers use. Bringing these teams to the threat. Modeling sessions is critical.

A

After all, these are the teams who are skilled at breaking into your systems, who better to validate your assumptions on what is secure. Another point to note here is that, if you are threat modelling in the existing system, a powerful technique could be to take the threats. You've identified ask one of the pen testing team to attempt to exploit those threats.

A

It's a great validation of your threat model. If you start identifying threats with the real world evidence produced by the pen, testing teams last but not least, include product owners, business, analysts and internal users in your threat. Modeling sessions.

A

These people know as much as there is to know about their data and they can help determine the boundary between necessary business function and security.

A

Now a few words of caution here, if you start workshopping with all of these stakeholders that only works best if they're completely familiar with the project and the solution, um I've been in a few sessions where that hasn't been the case. And, uh frankly, it's been chaos.

A

So if you need to arrange some preparatory sessions to establish a baseline amongst all these individuals, first always make sure that you have some documentation or architecture diagrams available that detail how the system works to refer to during these sessions.

A

Also, if you're threat modeling early early on with that t, this set of stakeholders and you've yet to define the architecture then just prepare to draw a lot of mock-up architecture diagrams.

A

Sadly, threat modelling is really done best in a room with a whiteboard. Unfortunately, as we all know, it's just not possible right now.

A

Additionally, if you're at a large organization, it's actually likely that your stakeholders will be globally distributed anyway, so video conferencing it is, and with that you'll be at the mercy of the collaborative tooling that you're allowed to use by your organization, so be wary of that. When you're planning your sessions.

A

Finally, how does one actually perform a threat model? Well, here's the basic threat, modeling processes defined by adam shostak and adopted by os. There are four simple steps: what are you building? What can go wrong once it's built? What should you do about those things that can go wrong and did you do a decent job of analysis?

A

So, for the first thing, what are you building steps all about information gathering and context, setting understanding the nature of the data criticality of the system, what the architecture is and how it's provisioned and maintained?

A

The second step is obviously the point at which vulnerabilities in the system and resulting threats are identified and crucially, prioritized following which you're going to create a backlog of mitigating security requirements, new architectures bug, fixes and so forth. In order to answer what should you do about those things that could go wrong?

A

The final step is to identify whether you did a decent job of your threat model, so you're going to review the outputs from the previous three steps, you're also going to incorporate external sources such as pen testing. At this point, once a threat model has been established for a system, it's this step. That's repeated just to make sure the answers to the first three questions are relevant to the system at the current point in time.

A

That's quite it's all been quite abstract so far, but uh what does this threat? Modeling process? Look like for kubernetes?

A

Well, the four c's of cloud native security are cloud clusters, containers and code so to secure your full stack. You're gonna obviously need to consider the configuration of the cloud service provider, the cluster, what the container is comprised of and how they configure to run and the security of application code itself.

A

Now we can broadly address the security of the latter three through the design of the cluster and the ci cd pipeline and regardless, regardless of whether you're a small or large organization, it's good practice to provide standardized components for application teams to consume, so you'll provide a cluster for them to use or a means to provision and scale their own clusters.

A

You'll define the baseline configuration of that cluster and you'll place some restrictions on what that what applications can do in terms of configuring that cluster you'll also obviously provide those application teams with a ci cd pipeline, used to build container images and and deploy those to your to your kubernetes cluster.

A

So for each of these patterns, you're going to end up creating reusable threat models and those threat models are going to generally going to be along the lines of cluster provisioning and scaling cluster runtime cluster configuration and then threat models for the application. Ci cd pipeline and the output to those threat models will define the architecture of the standardized components and patterns your application teams will use this all raises a really important point.

A

Doing a runtime threat model is not enough. You need to ensure you think of the architectural solution from an end-to-end perspective.

A

You need to ensure that you're truly thinking through things through right from the client experience through deployment to help operation stuff utilize. Your your cluster solution.

A

You can have the most secure system in the world at runtime, but if it's exploited, because you forgot about supply chain security and and deploying the system in securely, then uh you've wasted your time. You really have to think this through from an end-to-end perspective, so defining what you're, building any good architectural diagram will do. You can also use open source tools like cloud mapper to get as much info as possible on your environment, but just to mention data flow diagrams here.

A

Data flow diagrams are really important, really useful for breaking down what you're, building into flows, processes, trust boundaries and stores within a system, and once you have that depiction that makes a technique called stride which I'll describe in a couple of slides. That makes it a lot easier to do so if your architectural diagrams aren't fit for purpose.

A

Consider drawing a data flow diagram. Instead, you can draw data flow diagrams of a specific interaction like kubernetes, pod launch or the whole system, like this depiction of a simple ci cd again, you may need to create multiple diagrams just to reflect uh the end to end the end-to-end solution.

A

So now you have these architectural diagrams and you have all of this information.

A

You need to start identifying what can go wrong now, if you're experienced in the threat, modeling and security space then brainstorm to your heart's content. Annotate your architecture diagrams, make copious notes as you. Let your imagination run wild if you've not threat modelled before it will be helpful to use a straight framework stride and pass pasta are two examples.

A

So strides in acronym stands for spoofing tampering, repudiation information, disclosure, denial of service and elevation of privilege.

A

One can use stride to characterize and identify the kinds of threats that affect processes, data flows and stores within your system and their most common and stride is most commonly applied or used in conjunction with data flow diagrams because, as I said earlier, data flow diagrams and diagrams that do explicitly display those processes, data flows, stores within your system.

A

There are more other, more explicit sources of threats, for example, the mitre attack matrix and also reverse engineering benchmarks can also form a valuable source.

A

One warning a thing to bear in mind is that you need to define when to stop identifying threats. If you're, not careful, you can threat model in an endless spiral of despair, kicking out every issue out there.

A

So when do you stop if you're experienced in this space and everybody is familiar with kubernetes and cloud-native solutions, you can time-box your activities because you'll know roughly when you're done. Otherwise, if you're threat modelling for the first time, if you have one really good stride, attack per element represented in a data flow diagram. That's an okay first pass.

A

Finally, when you're identifying threats, there are a great a wealth of great talks and existing threat models available within the community, and so it's no surprise that I'm gonna talk briefly about cncf financial user group protectors.

A

So, in conjunction with other members of the cncf financial user group, we treat modeled the whole kubernetes system by asking what can go wrong. We looked at attacks of the api server, central d, cubelets, exploiting aspects of cni or csi. We looked at compromised workloads in the various permission, permissive configurations that could be accidentally granted. We also looked at some cloud: prime provider, integration and meta data. Api file, attacks- and these are tech. Trees are open sourced in github for your use now.

A

Attack trees are a highly useful threat, modeling tool, and you should use them whilst identifying your threats. They provide a formal, methodical way of describing the security of systems based on varying attacks.

A

Basically, you represent attacks against the system in a tree structure with the goal is the route mode and different ways of achieving that goal as leaf notes, attack trees were coined by bruce schneier in 99, and we've got a brief or simple example of an attack tree to the right here. It's simplified for the point of explanation.

A

Attack trees can be quite large and squarely difficult to present on powerpoint slides. So we've got this approached version here, so to start drawing an attack tree. You start with a negative outcome that you want to avoid. So, in this case, it's at the top and it's malicious code in the workload you subsequently ask how it could be achieved to flesh out the rest of the tree to explain the key on the right.

A

The shape of each node dictates whether the node is an and or at all, and nodes require both conditions joining the node from below to be true, whilst all nodes indicate that what lies beneath are multiple ways in which the node can be realized.

A

So one way to have malicious code running as part of the kubernetes workload is to deploy a poison container image.

A

And if you look below that node in order to deploy that poison container image, you need to poison the image within the container registry and the image needs to be deployed, and in this case it will be deployed as part of a normal release going further down the tree. In order to poison the image in the container registry, you will obviously need to obtain the image, pull secret and you're, relying on that pull secret, having right and overripe privileges within the container registry.

A

Finally, there are actually three different ways in which you could um obtain the image pull secret. You could make a commentators api request or you could use a running container with host file system access, thus accessing the password on the host or you can obtain it from a cubelet using the unauthenticated port, for example, now attack tray trays are useful tools for collaboration between dev teams and security.

A

Security teams, who aren't necessarily okay with all the cloud native technologies, often recognize the high level outcomes that must be avoided, such as information disclosure, um uh extra data, exfiltration and so on.

A

When these, when the when security teams are faced with new technologies, they don't know exactly how they're achieved, but a tech trees are a great way of fleshing this out with with technical detail below and helps establish a common point of reference between development teams and security when discussing the overall security posture of the of the system.

A

Now, if you don't want to use the tech trees, as I said, create tables or annotate your own diagrams, there's a threat model of a cluster at runtime. It's not pretty, but it's perfectly valid. Just labeled all the things that could go wrong in an architecture: diagram don't get too precious about your diagrams and architectures. You will be redrawing them and updating them countless times as you build your threat model.

A

So initially, don't worry about your connections being a little bit off because you'll be redrawing those lines soon enough now you've got a whole host of threats identified. Well, what are you going to do about that? Well, to define your control, sets, evaluate and prioritize your threats. Then map controls directly onto them.

A

You could end up with a particularly busy architecture diagram like so or you could annotate your attack trees by breaking different branches with an appropriate control. We'll talk a little bit more about the temperatures later. So when you look at selecting controls, what controls do you have to choose from? Well, frankly, there's loads, here's, a here's, a quick menu of controls, just like any good menu, don't order everything it will make you sick, we'll talk about how you minimize the number of controls you need to implement later so in the networking space.

A

If you're concerned about the compromise of a container and subsequent pivot across the network is kubernetes network policy for starters, if you're running a service mesh, then there's the ability to have the ingress and egress rules and access controls between workloads and layer, 7., initially you've got infra firewalls, whether they, whatever they may be, called for your cloud provider, be it security groups or kneckles or simply just firewalls.

A

These will provide you with networking restrictions at the node and sublet subnet layer level at layer, 3 and 4..

A

Also, some container native, ids and ips systems can now be configured to restrict the network connections made by running containers well worth. Considering what controls can you use to constrain a compromised container at runtime, maybe one that would attempt to break out onto the host?

A

The first thing is just to get the basics right, define a secure security context for your pods and containers. It's all the usual stuff run as non-route, then allow privilege escalation, drop, all your linux capabilities and use app, armor profiles or sc linux. If you can container compatible, ids and ips systems are pretty prevalent too, to use them ones like falcon.

A

There are also paid solutions that offer a lot more capability than just runtime protection, with container image scanning and mission control at all. So if your oil likes to throw money at things, then these are well worth using just finally, over the last few years, sandbox and isolation technologies have started to emerge, their implementations differ, but the goal is the same: it's to isolate the running workload from the host's kernel as much as possible.

A

The three that are currently the most prevalent are cata g visor and firecracker. It's a space to watch with interest.

A

There are currently a few limitations to investigate before you dive into using these for everything, for example, service mesh compatibility so, whether accidentally or maliciously, your users are going to misconfigure your clusters, workloads or just accidentally break things again. Getting our back right is one of the basics. It's not easy, though, but thankfully there are plenty of open source tools to help on that front.

A

Make sure you enable the kubernetes admission controllers, uh the cis benchmarks have uh recommendations on that front on what you should enable and finally, no talk on kubernetes, our back or policy is complete these days. Without mentioning the open policy agent provides the ability to enforce custom policy and appears to be the future in the pod security policy space.

A

There are a couple of implementations you could use as admission controllers, so kubernetes native implementation in the form of gatekeeper. You can still use a plain opa to enforce your your other custom policy. If you say wish.

A

Now the majority of the controls that I just mentioned are to protect your cluster from a compromised container. It goes without saying that the most important element in preventing this is embedding security within ci cd, which is a torque in itself, as a wealth of information on security checks and tooling. Each stage of obtaining a build pipeline for just a note of increasing importance and focus, particularly within financial services, is the ability to attest that a container image you have built has been subject to all of the security tests implemented as part of your pipeline.

A

This involves collecting metadata from each stage of your pipeline and validating it at cluster admission time. Graphic is paired with criticism and toto are the open source technologies to consider in this space.

A

So you might read all of that. You might have a threat model with loads of threats in it and then you might pull together. All of your controls decide to enforce the defense in depth and end up with this an example cluster. From the infra perspective, you can see how the infrastructure controls segregate nodes, allowing one to enforce controls on networking at the infrastructure level and theoretically segregating the cloud permissions. A workload can access at the instance level, that's theoretical, we'll touch on that later.

A

The architecture gets even more complicated when you look at all the services and configuration that will need to be configured within kubernetes itself.

A

My advice is not to do all of this, particularly straight off the bat, unless you have a really really good reason to and your threat model will justify it instead. Start simple after complement of these complementing control sets, there's absolutely no need to implement an entire category of controls to demonstrate defense in depth.

A

Your security architecture needs to be as simple as possible to mitigate the threats. You've identified to an acceptable level bear in mind the more security the more complex your security architecture becomes, the more automation you need to provision to both provision and automatically test clusters, in effect, there's the ability to provision a cluster supporting services and run a full suite of tests from just one push of a button. It's the only way to ensure that a complex solution is maintainable and scalable.

A

So when choosing the minimum set of controls, risk has to be the determining factor: prioritize the threats in your threat model according to the risk they pose.

A

That will help you go from a lengthy list of threats to understanding where to start first, which threats are significant enough to eliminate at the starting set of controls required at this point I'll again extol the virtue of the tech trees. There's controls you devised, labeling and plotting them onto your attack. Trees as a means of breaking the branches, are a great way of showing you how your security controls can theoretically combine to stop attacks.

A

They'll also demonstrate another means of defense in depth, not by doubling up on controls for a certain category, but showing how seemingly unrelated controls can mitigate attackers from working their way through your system.

A

Now, automated testing is often overlooked when implementing controls or codifying controls just like in mainstream software engineering, you wouldn't dream of releasing code without testing. So too, with codifying security, you must write tests to validate the efficiency of the control or the efficacy of the control.

A

If the first time your control is tested by the attacker is at run time, you're in for a bad day, so the only way to validate control implementation is to do so repeatedly using automated tests.

A

In some cases, you may be able to focus on implementing the controls in such a way that you can automatically test them at runtime. In addition to the sdlc, this gives a continuous security monitoring capability. It's like the holy grail of automated security.

A

It's important to note you test the threat to be mitigated, not the specific implementation you have coded for your patrol, in other words, check to make sure you can't access your s3 bucket via http, rather than validating your code is as you expected, while implementing it. This is a common mistake, often encountered by junior developers who test their implementation or the happy path, rather than truly checking the intent of their code.

A

Finally, in the test space, one potential project to be aware of is proposal, native security to secure kubernetes end-to-end via configuration and codified controls tests for the department of defense.

A

Earlier in this talk, we discussed the importance of ensuring that you threat model from an end-to-end perspective and ensure that the sock and other operational teams are involved.

A

Well, taking this to its logical conclusion, we must implement controls for our threat models and ensure we implement any appropriate detective controls with our soccer teams, also ensuring we have uh suitable, automated testing in place.

A

After all, what good is a detective control if it isn't written appropriately to genuinely alert a sock team, member or process so step? One? Do your threat model told you how to now off you go and do it step two reproduce the attacks identified repeatedly against test clusters and gather the signals step three take these signals and work with the sock to configure their their seam tool.

A

You may need some training at this point. You may want to use your attack trees, so your sock has situational awareness in order to understand how the signals they're receiving relates to an attacker's path through your system and so forth. You'll probably need to review audit configurations during this step, just to make sure logs are neither sort of too full or too slim.

A

Finally, you want to rerun your test cases and make sure the stock lights up, and you want to keep iterating on this. So as your system develops, you'll identify new threats, you'll identify new detective controls, you'll need to keep going back to the sock.

A

So if your organization is new to cloud you're in the middle of a kubernetes implementation- and you haven't thus far engaged the compliance and infosec team be aware that they are likely to be in a state of absolute shock as they evaluate your architecture for the first time, you're going to need to approach them, prepared, scaring them straight into a no is a real risk here.

A

So come armed with your security, documentation and threat model show them. There's some precedent, followed in terms of standards adhered to, like the cis benchmarks. For kubernetes and related test results from qpench also note if you're in gcp is that google have open sourced gke patterns that demonstrate adherence to pci dss, and, I believe, that's the subject of another talk here at qcon again, if you're, following this pattern or using this pattern, you're demonstrating precedent and adherence to standards, you can also get one step ahead of the curve by mapping controls to your compliance requirements.

A

Thus, your tests prove your compliance and you move the assessment away from something that is just checklist based.

A

Realistically, though, not all requirements bestowed upon you by your organization will be testable or even applicable to cloud native. So if policy rewrites are required, then use that as an opportunity to write policy in such a way that compliance can be demonstrated with automated tests.

A

She's going to finish with a couple of gotchas, so I mentioned node segregation. Earlier on. As a as a tool for implementing security, you can put firewall rules around your different nodes at the infrastructure level. You can also make sure that your nodes have different instance permissions so that your workloads are are not able or only able to assume a minimal set of roles. So you could keep privileged roles on different notes.

A

Well, that's still too much thinking along the happy path, so tim, mulchler and greg castle gave an excellent talk at cubicon san diego, on how using node segregation does not guarantee security within a kubernetes cluster. Their demo showed attacks that included being able to reschedule malicious workloads on other nodes.

A

These were historically possible and shouldn't be possible now, but the talk is worth revisiting and taking into account and deciding how important node segregation really is within your security architecture.

A

So if you run a service mesh in your cluster, it will need some linux capabilities that would not be authorized if you had implemented a restrictive pod security policy so, for example, with istio. If you have an installation that does not use the istio cni plug-in, which is still in alpha, then istio init containers will be run in each of the application pods, and each of these require nat admin and net raw linux capabilities.

A

Now, if I recall from about a year to 18 months ago, the istio containers back then ran with, I believe, maybe the privileged flag and root. Don't quote me on that, but it appears that things are much better now, regardless, you will still need to relax your pods security policies to allow these capabilities.

A

Now, the eagle-eyed among you will recall liz rice's talk from qcon 2019 in north america on the dangers of capnet raw. However, the attack from that talk doesn't succeed within istio service mesh, hence the asterisk.

A

So regardless, if you want to lock this down, the solution is to implement some custom policy using opa uh within a allow list for istio containers.

A

So, specifically, largest gotcha of them all is that large regulated organizations with existing on-prem infrastructure tend to take that mindset into the cloud so be wary and keep in mind that introducing cloud native and kubernetes into a large regulated organization requires as much of a cultural change as a technological change.

A

As you all know, large regulated organizations like financial services organizations have numerous compliance requirements that results in security architectures that place special emphasis on a few areas: strict change, control, standardization of security architectures and extensive logging, audit and protective monitoring requirements.

A

Unfortunately, the byproducts of that focus over many years within an on-prem environment have resulted in the following practices: heavily manual change, control processes which are slow time and resource consuming our reliance on detective, rather than preventive controls within security architectures. So an example of that would be the manual change control process that involves admin, click, ops and then reliance on the sock to detect any misconfiguration made by that admin after the event now, neither of these practices are particularly secure, maintainable or scalable within cloud, and so obviously red flags to be wary of.

A

As you start, your migration to kubernetes or cloud.

A

So, in short, I know we're running a little bit over threat model. Do it early on get everyone involved, keep it up to date, draw attack, trees, they're, really useful. They break down high level threat outcomes into discrete steps, providing a great resource for security and for writing tests. They allow you to determine the controls needed to secure your solution without over-egging the cake they provide. They also provide a great source of situational awareness for the sock, make sure you apply and test your controls in an automated fashion, remembering to test the threat.

A

The control mitigates, not the implementation of the control itself and once you've done. All of this you'll be able to run these automated tests against test infrastructures so that integrating with the sock should be relatively easy.

A

So we're finished, with the obligatory hiring slide and thanks for attending the talk.