Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Putting Hackers Breaching Your Cluster In Automatic Quarantine - Ziv Nevo, IBM

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Putting Hackers Breaching Your Cluster In Automatic Quarantine - Ziv Nevo, IBM

Speakers: Ziv Nevo
Engineers can’t really prevent hackers form eventually breaching Apps. It is not a question of IF but of WHEN. And unfortunately, a question of how much damage was done to our or our users’ resources, data and reputation. This does not happen only to small Apps and companies with small budgets and limited resources but to huge companies and government agencies (see SolarWinds attack). The solution - automatically isolating attackers when they breach one of the Apps in your cluster (or the App you develop), keeping the rest of the cluster’s components safe. This session will present a survey encompassing many commonly used cloud native apps, engineers all love and need (like Prometheus, Kafka, Jenkins, ClearML and much more) and demonstrate the built-in vulnerability most cluster deployments exercise and how to secure it. State of the art practices leave several, rather easily breached, back doors in many clusters. We will deep dive into several real-world scenarios and see the simple, yet very often missed, blueprint for making our cluster or our App-users’ clusters much more malicious-resistant.

A

So I'm Dave I'm a senior research scientist in IBM research in Israel, and this talk was proposed by my colleague, shmulich voimovic, who unfortunately couldn't be here today, and the title of this talk is putting hackers breaching your cluster in automatic quarantine. So you can see this is a talk about security and I was told that talks about security should start with scaring the audience that something frightening is going to happen. Sir, here is a scary thing. Really, it is I have some of these in my backyard and they are nasty.

A

They are very quick and when they bite it really hurts so don't mess with these creatures. um However, I got used to them, and now what really scares me is that too many Cloud applications do not apply. Now. That's scary, you don't seem scared enough. So let me explain why this is so scary, mainly because without a gas control, your application is exposed to many many attacks and, for example, reverse shell attacks data leaks.

A

If you don't control, what gets out of your cluster, everything can get out can get out of your cluster and it is very likely that there are hackers that may already be in in your have control on on some microservices in your application.

A

So previously it was thought that if I protect my English good enough, this is well. This should be enough, but there are Insider threats there always have been, and we know that social engineering and spear phishing attacks are getting more and more sophisticated, and this also allows hackers to get into your application, and those of you who attended the security cone may have heard the very frightening number of 742 percent rise in supply chain attacks in the last three years. So this is really scary.

A

So, basically it's not a question of if it's a question of when your application will be hacked and you should be prepared and at least protect your egress, so data doesn't get out so easily.

A

So this is the customary slide with some recent attacks that maybe it could be prevented with tighter egress control.

A

Oh- and there is also compliance so and these days you have standards so and the nist sc7 tells you that you should deny network communications traffic by default and allow natural Communications traffic by exception and manage interfaces, and only those system, connections which are essential and approved order and are allowed- and you may say: okay but I, closed the interest side matte. This doesn't mean that a micro service of yours cannot talk to the outside internet, and so you must control also its igles.

A

So why is it that Ingress is so exposed? So first we have lack of awareness, hopefully yeah. Now we are aware of that. This is something that you should think about and do something.

A

Then people tend to trust, Cloud vendors so to to have the protection to put the protection for them. Well, this is usually a bad idea.

A

Cloud vendors can't really close all your egress or otherwise, things are not going to work and they cannot prepare for all use cases now.

A

A third reason might be that you say to yourself: I didn't, do much I just took WordPress and MySQL and Prometheus and go fun and put them all together. I should be okay, they know what they are doing.

A

They know what what they are doing, but this is not enough, and I will have more details about it soon.

A

Is considered hard to configure and to maintain- and this is actually true- this is not reveal, and also sometimes you do have rigorous control and you do have you do limit your connections, but the configuration is not really what you meant. So the configuration is wrong, and maybe you are not even aware that this is the case.

A

Finally, in a development team, it is really unclear whose task is it to put the Protections in place? Is it the development people Developers? Well, they usually don't care much about security. They just want their code to work place. Is it that devops people well, they are not really aware of which connections should be in your application, and so they may not be the right people, maybe the security people.

A

Well, they don't really. They know the rules, they are not always sure how this all should be applied. So my claim is that for the last three bullets, actually automation can help. You can help us here and basically this is John. This is the agenda for the rest of my talk. uh I will show you why trusting third-party components may not be such a good idea and I'll show you how automation can help just one word before I continue into that. So where is the best place actually to control egos?

A

You may have several choices: maybe the cloud provider firewall or the VPC Network accounts or security groups, maybe the classic. Maybe you have an egress controller for your cluster um well I, think that it is best to put Eagles as close as possible to where your data lives all the other places.

A

Well, it may be too too late to stop data leaks at higher levels, because they are not specific enough for the kind of Communications that are really allowed, so in kubernetes based application. This actually means that you want to control pod traffic and the way to do it in kubernetes natively is Network policies, the also a cni specific Network policies like psyllium Network policies or Calico Network policies, but these are cni specific, and if one day you want to migrate from one cni to the other, then you'll have to migrate.

A

There are also service mesh, specific Network policies like istia authorization policies. However, these are layer 7 only and consider less secured.

A

So now that we know about Network policies, let's see so we made a small survey of some common open source packages, so we were looking at 15 of the most popular Helm packages from artifact, Hub and Vietnamese, including databases, registry Event, Systems, all the package, all the packages that you like and imagine here, but not getting to names.

A

The good news is that most come with some index control, and this is good, although typically, you are required to put some flag some non-default flag, you have to put it on, but that's the good news. The bad news is that only four of these packages, out of the 15 that we looked at only four included some sort of legal control and to remind you, this includes databases. So if your database has no egos control might be that your ports can send data directly to a to the to the internet outside to hackers yeah.

A

It's not a good idea, so never trust other security, oh as we call it zero trust not to be too pessimistic. Here is one example where things are done right, so you can see here that the true stateful sets, in that case talk to each other in restricted in a restricted way and are connected to ultra involved only through UDP 53. So this is the only Ingress that is allowed. This is DNS. This is considered fine.

A

So you may ask well okay, so this is a very nice graph. How did I get this this graph? How do I know if my package is doing things all right or not?

A

So this is what I said. Automation can help and there are tools to help you with this observability.

A

So here is a incomplete list of tools. These are the tools that I know about.

A

There might be some more and there are tools that can look at your live cluster and let you know the connectivity posture, so some of these tools can only only look at the actual traffic that flows in your cluster, so they may not be aware of hypothetical traffic that can go out of your cluster, but this is definitely better than nothing at least nowhere where you stand, other tools can actually analyze Network policies and tell you what is the allow traffic and what is the and what is denied.

A

So we have here tools like Reddit, ACS or Star clocks. We have cystick secure, cilium, Hubble, calicoin, Enterprise and NP guard, which is an open source project for which I I'm a maintainer.

A

um Now, entry guard can also look at your repo and tell you what is the expected connectivity posture for your application, given the yaml Manifest that live in Europe, so this actually supports shift life methodologies, so this is about knowing where to stand, but how do I get to the place where my connectivity is done right in here? Automation can help once again, automation can help you synthesize Network policies, and there are two kinds of tools here.

A

One kind is tools that look at the actual traffic in your live cluster and use it as a baseline for synthesizing Network policies, so we're at ACS and syslig secure can do exactly that.

A

The other type of tools is tools that are analyzing. Your yaml manifests and understand. What is the required connectivity from disable manifests and when they know their report connectivity, they can synthesize nephron policies that will only allow this required connectivity and nothing more so NP guard is all once again one one of these tools and the technology used by npguard.

A

He is now going into a stock rocks, which is the Upstream foreign.

A

Philosophy, as you can see, is doing things on the left is configuring the connectivity on the left and it is, and the vision is made of two phases.

A

So there is the analysis and synthesis and synthesis phase in which Network policies are being synthesized, and this is proposed as a network configuration for the user to review now, net NP guard will not just provide yamlers to the users and say well. These are network policies. Take a look. We want more than that, so we can provide the connectivity maps that we've seen before we can provide connectivity diffs.

A

We can tell you what what is changing in terms of connectivity, not a syntactic diff of yamas, but rather telling you which connections will no longer be available after you merge this pull request and which connections will be added, which connections will be modified and there is also the there is also the issue of organizational policies so you'll see somebody tell you that some protocols or some ports should be blocked, or maybe some there is a deny list for outside IPS, and this can be specified as roles and they can be used.

A

Both in the synthesis part, so we don't synthesize anything that is not compliant and they can. They will be rechecked at the review process. Even if the user chooses to add more connections, because maybe the synthesis missed something, then any change to the network policies will be rechecked against the organizational rules and only when the neutral configuration is compliant. Only then can it be deployed to to the live cluster.

A

So let's go ahead and see a demo of how this can work. So this use case is a Greenfield use case. So we have no network policies at all and I decided I decided now. I want to have some protection, so I will initiate this analysis and synthesis step workflow.

A

This will generate Network policies, open a pull request with the generated Network policies and other workflows will annotate these pull request with the connectivity map, the connectivity, div and we'll check that the connectivity is compliant with some high level rules. Let's see a demo.

A

A

A

Yeah, okay, so this is the online boutique example made by Google a very popular example, so this basically this is basically an online shop for vintage products, and here is the architecture for in this application. It is made of 11, also microservices, that talk to each other in a non-trivial way.

A

However, this application comes with no network policies, so no protection at all and, for example, the front end here. If a hacker can get from the internet into the front end here then the front end can act, can directly access the payment service without going through the checkout Service First.

A

Moreover, the payment service can actually talk to the outside internet. There is no protection at all here, so data can leak from the payment service, all the credit card numbers and so on.

A

So we want to apply Network policies, so we have your integration. We integrated it with GitHub actions. In that case there are also Integrations with tecton tasks and can also be consumed. The components can also be consumed as Docker containers. I will go to the synthesis workflow here and I'm going to. Maybe we first take a look at the workflow and see what we have here, so basically, what this workflow is doing.

A

It checks out the code and the application code and the manifests and called the npguard netball synthesis, get abduction and we then use the artifact or the synthesized Network policies that were produced by NP guard and commit them to a branch and open a pull request. So this is how it works.

A

A

Let's hope this is going to work in the last five days, I've seen a remarkable number of failing live demos, but even on the edge, it's me and the centipedes okay, so yeah it runs. It should take around 30 seconds.

A

And once again, this checks checks out the application, manifests it analyzes the Manifest to understand what is the required connectivity and synthesizes network policies to allow only this connectivity okay, so it is done and it has opened the pull request for me, which is very nice and let's see what it has done, so it actually added one file with network policies.

A

So we can see here we have one network policy per micro service. So for the checkout service we have this network policy saying that it can connect to the image service, the payment service, the product and so on, and so on and so on, and it can get Ingress from the front end and the email service here can get connections from the checkout service and so on.

A

But no one likes to look at Germans right, Amazon, not easy to understand so, hopefully, I should have here if I, refresh, I, guess yeah cool, so I got a few pull request comments here that, let me know what is my current status, so here's the all the allowed connections I, have here a pull request comment with all the allowed Connections. In my application now so I can set up the checkout service.

A

The front-end and the recommendation service can connect to the product catalog service by using TCP 3550 and the checkout service, and the front end can talk to the shipping service using this part, and we can also see that it is only the front end that is exposed to the outside world, which is called. This is really what we want and we can also get a connectivity map like that, which is much easier to understand, but tables are also useful, especially for audits, okay, so that's cool. What else have you got.

A

So there is also this diff here, so another pull request comments. That is telling me the difference between the previous state of my application and what's in this pull request now, since the previous date was that everything was allowed, all connections were allowed in the cluster, so many many removed connections when I apply this network policies. So you can see here that many connections are no longer available, both in cluster and also from outside of the cluster to the cluster. So all these connections are no longer available, which is good. This is what I wanted.

A

I wanted to remove many redundant connections and leave only the connections that are really needed.

A

Finally, what I have here I have here if I started a few high level rules.

A

Another pull request comment: this is checking that my connectivity is compliant with several high-level rules, so my ciso, for example, told me that no FTP is allowed in the cluster or no telnet, no SMTP, no non-secure protocols and all these walls actually pass, which is nice, but I have here a failing goal. So there is another rule that every micro service that would like to access. The payment service should have a specific label, and this rule is failing here and we can see in the details here.

A

The checkout service is trying to access the payment service using this protocol in Port, but it doesn't have the required label. So you may ask: why did synthesis? Allow such failure? Well, the reason is that I didn't provide the soul to the synthesis to the synthesis workflow, because I wanted to show you a failing goal and this roller actually blocks this pull request from being merged, which is what I want I don't want.

A

um I, don't want any network configs that are not compliant with the rules. So let's get back.

A

A

Okay, so once again, this was like a green field, use case, I started and and I had an application with no network policies and I started. The analysis and synthesis workflow which generated Network policies, put them into a pull request and other workflows put a pull request comments to show me: the connectivity map, the connectivity, diff and the organizational policy validity.

A

Sir, to summarize, the German English is not enough. You should also always secure egress and the best place to do it is at the microservice level.

A

It is important that you will always know your connectivity posture and there are tools to get it. So I mentioned the list of tools. There might be others, but do know where you stand.

A

Also. There are tools to help you with a synthesizing Network policies suggesting Network policies.

A

These tools may not be perfect, yet they you have to look at what they produce and make sure that they don't miss any required connections, but they will take you to a good place to start with, and it is possible to integrate both synthesis and Analysis into your cicd Pipelines.

A

So you can have a shift left methodology that will make sure that your connectivity is secure.

A

That's basically it one more advice. Do well high top shoes when centipedes are around that's important.

A

And if there are any questions, I'll be happy to answer.

A

Questions I wave at me: if you have questions.

A

Okay, thank you very much, then.