Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hey, Did You Hear About This New CVE? - A Vulnerabilit... Andrew Lytvynov & Alexandr Tcherniakhovski

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Hey, Did You Hear About This New CVE? - A Vulnerability Response Playbook - Andrew Lytvynov, Independent & Alexandr Tcherniakhovski, Google

Kubernetes provides a lot of great ways to harden the security of your infrastructure. But despite how much you have it locked down, things will inevitably go wrong. Hope is not a strategy, so you need a contingency plan. In this prescriptive talk, you will learn how to prepare for complex vulnerabilities (like the Intel speculative execution or Go HTTP/2 DoS vulnerabilities), mitigate them and clean up afterwards. We’ll share our experiences with the GKE fleet, dealing with newly announced vulnerabilities, and lessons learned balancing both security and reliability of workloads. You will learn how to develop a response playbook for vulnerabilities. You will also learn about the tools that Kubernetes provides to help make your vulnerability response less hectic, such as audit logs, network policies and RBAC. This talk will not cover live attacks (attackers in your infrastructure).

https://sched.co/ZerA

A

Hello and welcome to our talk, my name is alex chernicholski and I will be co-presenting today with andrew ludwinov for a number of years andrew and I have worked together on gke's security team, I'm still on the team and andrew recently joined gravitational as part of our on-call duties andrew and I mitigated multiple security incidents and we believe that we have learned something from that experience and would like to share it with you today.

A

But before we start, we would like to tell you a few years back.

A

In 1935, boeing model, 299 could fly faster, cover longer distances and carry more weight than his competitors.

A

Therefore, the flight competition, where the winner would be awarded a defense contract was supposed to be just a formality. However, model 299 crashed during the elect during the dam of flight.

A

The investigation showed that the crash was due to the pilot's error. This is despite the fact that the pilot major proper p hill was one of the most experienced test pilots in the usc air force at that time.

A

Furthermore, the investigation concluded that the plane was substantially more complex than the previous models. It seemed like it was too much of a plane for one person to fly.

A

Boeing did not get the contract and almost went bankrupt as the result. Nevertheless, several test pilots were convinced that the plane could be flown safely and precedent was further investigation and testing.

A

The result of their investigation was an incredible piece of technology. A flight checklist in the end checklists allowed model 299 to fly 2.9 million kilometers without one incident fast forward to apollo 12 lunar landing, and we see that checklists have become a norm in the aviation and space exploration industries.

A

Furthermore, the value of checklists is now well understood in fields other than aviation.

A

The who surgical safety checklist, which contains 19 items, has shown significant reduction in mortality and is now used by my by a majority of surgical providers around the world.

A

So what do checklists have to do with preparing for a security incident?

A

Gt security team is frequently engaged in mitigating kubernetes vulnerabilities.

A

What we have learned from this experience is that, even despite having access to highly skilled engineers and good tooling, we almost always feel like we've missed something very basic.

A

This is when we thought about creating checklists note that we don't want to trivialize the process by definition. Vulnerabilities are unpredictable and complex.

A

Thus, no checklist will replace the need for deep technical expertise and creativity typically required during the mitigation process.

A

Please note that, in the context of this talk, we consider the discovery of a vulnerability within any of our production components to be a security incident.

A

Other security teams may only declare security incidents only after some evidence of an active attack has been produced before sharing some of the pre-incident checklists that we created we'd like to point out. Some of the challenges involved in handling incidents in kubernetes.

A

Fundamentally kubernetes is a distributed system, which implies that a successful attacker will most likely need to traverse through several rings of your infrastructure, to reach the ultimate goal which we assume to be users data.

A

Therefore, the task of investigating an incident could, in many cases be reduced to correlating logs on multiple systems, thus tracing attackers path from exploiting the initial vulnerability to their ultimate target.

A

However, as you can see from this list, we have many challenges ahead of us.

A

We are about to present four pre-incident checklists that we believe will simplify the task of assessing your readiness, as you can see, that each checklist corresponds to a layer within kubernetes infrastructure.

A

As we go through this checklist, they will most likely appear to you as some general hardening guidance, which you probably already heard before, and indeed the best strategy for preparing for an incident is to reduce the chances of it happening in the first place and hardening of kubernetes infrastructure is the best approach for accomplishing this in the context of hardening. You may have heard about kubernetes ces benchmark, which is, in fact a checklist that is designed to assess the overall security posture of your kubernetes environment in gke.

A

We take ces results very seriously and strive to get to 100 compliance, so if you're already using say as benchmark, then that is great you're already taking advantage of the checklist mentality.

A

In this talk, we'll take a slightly different view on hardening.

A

We will zero in on the configuration settings of kubernetes that we believe not only will reduce the chances of a compromise, but are also particularly valuable in generating signals indicative of an attack, in other words, the checklists that we will offer once implemented, are meant to force attackers to generate signals which we could detect and react accordingly.

A

The last thing that you want to learn when mitigating an incident is that the vulnerable service has sensitive cube api permissions, which which it actually did not need. Essentially, this is a gift to attackers, therefore review our back profiles of all your critical services and ensure that they are assigned the minimum set of permissions required.

A

How do you do that, though? Well one way to find this out is to enable cube api server, audit log and observe the request that your service is making under normal conditions.

A

We also need to ensure that access to sensitive resources, like secrets, is audited for both successful and failed attempts. Remember attackers may not know if the compromise component has the permissions for these sensitive resources.

A

However, they will most likely try accessing them anyway, and this is our opportunity for detection.

A

Luckily, attackers can control which component of a service is exposed to a vulnerability. Therefore, most likely they will need to pivot from a vulnerable component, which may not be of any direct interest to them to their ultimate target.

A

In this act of lateral, traversal attackers will attempt to establish network connections to the surrounding components, and this is again our opportunity for detection.

A

If we know that components x and y within a service are not supposed to talk to each other, then such connection attempts are strong signals of an ongoing incident.

A

However, to collect such signals, we need to have thorough understanding of the expected network patterns within our services.

A

The second check on the list is very much related to the first one and assumes that we can recognize rogue network requests.

A

However, we still need to make sure that we are collecting exporting and retaining such network logs note that raw log records may not be sufficient. Remember that kubernetes in kubernetes internal ip addresses are not guaranteed to persist. Therefore, additional metadata may need to be attached to the to the network records, for example. Attaching a pod name will definitely be more helpful than just looking at some random ip address.

A

Kubernetes network policies is the mechanism that allows us to produce such logs. Furthermore, network policy via the logging mode could be also used to produce the network flow diagrams for your services.

A

Note that kubernetes default network policy does not natively support log actions. However, network policy engines like calico, do support this functionality.

A

One more thing for the network checklist make sure not to enable host network setting in the pod spec, unless you have very strong reasons to do so. We noticed that this setting is often used to simplify intra node communications.

A

However, in addition to removing the protection afforded by the network space isolation, the setting also makes network log analysis much harder, because source addresses are four components that are running in host network. Namespace will be the same: that of the host, thus making it more difficult to detect rock network connections.

A

Running containers is root, which is, unfortunately, the default is rarely necessary. Furthermore, such containers are most susceptible to escape attacks. You probably already knew that.

A

However, what we also learned is that this default also makes it harder to examine os audit logs at the end of the day, containers are still processes and when they all run as the same root user, it makes it more difficult to audit them.

A

Regarding linux capabilities, our experience has shown that the capabilities included in the default docker profile are too permissive. This default profile contains many capabilities that are not required by most applications. However, container escape attacks may utilize. This default capabilities to their advantage.

A

Hence we recommend dropping them unless your application has some very specific requirements from the incident preparedness point of view. When containers are running without these default capabilities, we expect that container escape attacks will generate access to night errors, which will be recorded by the linux audit subsystem.

A

This works to our advantage.

A

Allow privilege escalation is related to linux capabilities. When enabled it guarantees that the linux kernel will not grant any additional capabilities to the process once it is started.

A

I am personally a bit conflicted about the setting on one hand, this is a great way to ensure that your application, even under the coercion of an attacker, will not expand. It says its set of privileges.

A

However, on the other hand, security conscious developers may follow the principle of just-in-time capabilities and programmatically request and drop capabilities from the kernel. This is how creators of the capabilities envision them being used.

A

Unfortunately, this setting breaks such security conscious applications. However, in practice such capability aware, applications are rare and therefore it is probably safe to enable this setting on most applications.

A

Building docker images on top of this release of scratch is a well established security practice, since it reduces that tax surface of our containers.

A

Furthermore, disrelease containers make it more difficult for attackers to perform lateral traversal via exacting into the surrounding containers.

A

Distress containers also remove the need of solving a tricky problem of having to log all commands executed within our production containers. Let me explain this point.

A

We know that attackers would try to exec into the surrounding containers once they enter into our environment.

A

Therefore, we need to collect logs of all commands executed inside our containers. However, this is not a simple task. After thinking more about this, we realized that transitioning to distrolus could remove this requirement.

A

I recall that this release, containers or or containers built on scratch do not have a shell into which attackers could exact.

A

One last point on distroless containers in gke. We almost exclusively use go disrelease. Containers since go is our preferred language. However, when using digitalis containers for scripting languages like python, for example, attackers may still be able to execute scripts inside such containers. So please keep this in mind.

A

The second check, the link between the binaries running in production and the air manifests requires a little bit more background.

A

Let me first define what we mean by build manifest build manifests among many other things should contain the list of dependencies or libraries that make up the output.

A

If we take go mode file as an example, we can see that it qualifies as a build manifest since it contains the list of dependencies required to build a go artifact.

A

Of course, the versions of the dependencies are very important as well, since vulnerabilities are often applicable only to specific versions.

A

So the question is given a docker image, or rather docker image id. Could we tell if the packaged application depends on a problematic library?

A

Furthermore, during the mitigation stage, your incident response commander will for sure ask this question how many nodes- or maybe clusters are running with this problematic library.

A

To answer these questions, we clearly need to have a way to go from the image identifier to the build manifest, and you certainly don't want to start figuring out how to go from a to b during an incident.

A

The mechanics behind the linkage may depend on your built and deployment systems, so we can't go into the details here. However, we do encourage you to go through an exercise where you pick a production binary and try to answer a question on what libraries does it depend on.

B

So you have done your homework, checked all the boxes. You have your hard infrastructure with audit logs streaming, that's great, but what do you do when an incident actually starts?

B

Well, as you probably expect you just need to follow another checklist, there is one fundamental difference between the checklists that alex just talked about and the ones we'll look at next. The pre-incident checklists are trying to centralize all the power and decision-making in the hands of the single person who's following them.

B

The checklists for the incident itself, attempt to decentralize power and force you to communicate with each other. The the goal here is to split the problem into smaller pieces that smaller groups can tackle.

B

We will break down the incident into five distinct stages, and the steps for all of these will should be written down in your internal wiki when an uncle engineer gets notified about a new incident. The first thing to do is reproduce it and figure out the impact. So you want to know whether this actually affects your production environment, or maybe it's a false alarm due to some funky configuration that you use.

B

You also want to note what the blast radius is. So what can a bad guy do? How much of your infrastructure is exposed? How many of your.

A

Users are exposed.

B

And so on, a useful tool here is calculating a cvss score, which is an industry standard for rating security incidents. It gives you a score from 0 to 10 above how bad the issue is. After you have successfully reproduced it. You want to write down those steps that you used. This will be useful later on when someone else is rolling out a fix to production, and they want to confirm that it actually fixes the problem, and one other useful thing to note down is what are the symptoms of an active exploit.

B

Can you use any of the signals like logs or monitoring to later figure out whether this has actually been abusing your infrastructure?

B

Now that you have confirmed that the incident is real, it's time to start the actual response process? First thing you want to do is creating communication channels. The most useful thing here is a shared document that you can collaborate on with your teammates. It should usually be copied from some template that you have created ahead of time, so you don't have to figure out what sections need to be present.

B

It's also a very useful tool to onboard new responders and give them a latest information about the incident. Another channel. Another communication channel is a shared chat room.

B

This is good for just live coordination and some discussion, but any major decisions you make here should be recorded in the dog regardless and finally, if you are operating under embargo or have some other reasons to be secretive about the.

A

B

It's good to come up with some code word to refer to it internally without disclosing any details, but this is rather because there are multiple people involved in incident response, it's good to establish a clear hierarchy and chain of command.

B

So here are just some of the roles that you might define ahead of time and clearly communicate to people what they should be doing. The incident.

A

B

Is the main decision maker? This is usually the on-call engineer who responded to the incident originally and is responsible for driving it from start to finish. Communications lead is someone who will communicate externally to your customers or partners about the incident or any impact it might have on them. You might want to onboard some subject matter.

A

B

People who are intimately familiar with the part of the system, that's compromised, just to advise the unfix or maybe do the things themselves and then operations lead is an optional role. That applies when you have a large incident with a lot of moving pieces operations lead handles only the mitigation and rollout and offloads all those concerns from incident commander's plate and after all, this prep work we finally get to patch the vulnerability.

B

So this is pretty straightforward. You should make a short-term fix that stops the bleeding and closes the gap as soon as possible. You should also think about some longer-term ideas to fix this properly, or maybe you can fix the entire class of vulnerabilities that this represents.

B

After you have the fix in it's time to roll out, you have a choice here between your regular rollout schedule or a fast rollout. So this is something very serious.

A

B

Being actively abused, you might want to choose to forgo your regular safety checks and get it out the door much faster. Regardless of what you do. You should avoid outages, consult with your sres, your devops engineers, and make sure that whatever you're doing is not going to bring down the entire service and then after the reload is complete. You can finally call all clear and be done with the incident just kidding, there's more work to do so after you've fixed the problem.

B

You want to investigate whether it was abused so use your audit logs that you've set up before the incident use any other monitoring. You can have and figure out how many of your users were affected.

B

You do want to collect the list of users that were affected, but that should be a heavily restricted doc and internally. You should only communicate an aggregate number and.

A

B

If the number of people effect is not zero, you want to follow up with them and do the appropriate next steps and uh another useful thing to do is writing a postmortem. This is the same postmodern practice that sre is used for outages and it has been covered extensively in the past. So here's a link to a place. You can start learning about them if you're not familiar, but basically you write down what went wrong when it happened. What did you do in response and how can you improve in the long run?

B

I hope we have convinced you that checklists are a very valuable tool both before the incident as a way to prepare yourself and during the incident to make sure you have not missed any critical steps and remember you can always change them and adapt them. If there's something missing a step, you wish you had taken, add it to the checklist and save and the longer you use them the easier your incidents will become.