Cloud Native Computing Foundation Cloud Native SecurityCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Native Security for the Rest of Us - Tiffany Jernigan, VMware

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Cloud Native Security for the Rest of Us - Tiffany Jernigan, VMware

Your mission is to secure the vast tracts of land of the Cloud Native security landscape. Where do you even start?!? It would be preposterous to cover that whole topic in a single session, but we can at least map it out. The plan is to break it down into three key areas and review each in turn. * Platform - securing and upgrading our control planes and nodes; isolating compute, storage, and network resources; managing privileges and secrets. * User management and permissions - various ways to authenticate and authorize user access; leveraging tools like RBAC and Namespaces, and some common "gotchas". * Software supply chain - what that means; some actual threat models are; how to mitigate them. You will leave this session with a stronger understanding of the breadth and depth of Cloud Native security and resources to further develop your knowledge.

A

All right, this is going well so far. Everyone yay all right, so hi um yeah, I'm, Tiffany, Jernigan I'm, a developer Advocate at VMware, as my slide says up there um if you use Twitter and want to talk afterward or whatever, my Twitter is up there as well. Here's how more technology works, because sometimes this thing here fails as well all right cool.

A

So this talk is basically focused like this is a security conference but I'm hoping and assuming that not everyone here is like. Oh, my gosh I am a security. Expert I know like a ton of stuff. Hopefully someone here is newer to it.

A

um If not well, you probably know everything here already or maybe you'll learn something and that would be cool as well all right so first off, basically less is more the less that you have the more secure that you can be and I'm really bad at holding a mic doing this, because I haven't done this well so I'm!

A

Sorry, if I keep like moving around, you can't hear me so uh first off have a less code so where it's possible try to use unmanaged to use, managed services and existing like Security Solutions, already out there, instead of going and creating your own, basically, if you're, using something that is like off the shelf, maybe there have been a lot more people that have already been working on it unless you go and create your own specific security team to do it yourself, so chances are it's probably better unless you're, specifically focusing on doing that also give fewer permissions, for instance, only give necessary permissions, don't go and be like here, I'm going to give every single person admin access to everything and also avoid having long-lived secrets.

A

Also, you want to have fewer dependencies so that minimizes the tech service, so, for instance, maybe you'll want to use, say a disturbless image instead of using Ubuntu also keep up with whatever the latest recommendations are. If you go to uh the kubernetes docs, for instance, there is a security checklist there. You might not be able to see it too. Well, there I'll be posting.

A

Some of the slides later, but basically there's a link there that frequently gets updated with things so the kubernetes documentation it breaks down, Cloud native security into four C's, so there's Cloud cluster container and code as a protection. As a practitioner, you basically need to worry about hardening your infrastructure at every single one of these levels. You can't just be like I only care about one thing and not the rest of it.

A

So first I'm going to start talking about things with cluster components.

A

So unless you really need to run everything yourself, I suggest that you don't instead use a managed kubernetes offering so manage to manage services. They take care of a lot of the things for you, so you don't, so you don't have to do it yourself and therefore there might be a greater chance of like not making a mistake. Screwing something up more of security, vulnerabilities, Etc, so security and Harney is basically one of the primary value ads for cloud platforms.

A

So, as I was kind of mentioning, you need to worry about hurting for every single layer, so Harden the control plane basically only allow people to interact with the API server via the kubernetes apis. The less that you have running on your control plane servers, the less potential Avenues a attacker can exploit to gain access to your cluster, make sure that you use real TLS certs, such as say from let's encrypt and please do not be lazy and Skip TLS start verification.

A

There's this nice little flag that is insecure, skip TLS, verify it's basically as good as not having TLS at all. So there's also. This talk called pki the wrong way in that it's demonstrated how the end user ability to create client certificates for mtls can be leveraged to gain complete control of a cluster. So this, basically it demonstrates the need to restrict access to etcd. There is documentation in the kubernetes docs that basically tells you how to go about doing that, and basically at CD by default.

A

It assumes that anyone that can successfully authenticate with mtls has proper access, which isn't necessarily what you want all right, so um make sure that you use pod security standards to not allow others to have access that they shouldn't have so pod security policies.

A

They were the default so like there's a bunch of acronyms PSPs PSS, but um that was that's deprecated in 121 in favor of PSS there's also where you have the cloud metadata server, um if you're, using uh if you're not like running at, say just like completely on your own servers and locally, and everything like that, um and there are credentials and info that are given into the virtual machine. You basically never want to have your pods to have access to that, so there can be lots of kind of scary vulnerabilities.

A

That can happen if you allow talkers to have access to the metadata server. So someone could get credentials, for instance, associated with the VM service account and basically like. If the Pod can have access to this, it can get your node credentials and that might have you might be able to do things like registering to the cluster itself during like bootstrap. So a pod could put some amount of work, uh pretend that is a node and get workloads assigned to it, which is pretty bad so for upgrading.

A

uh If you're dealing with all this yourself, there's a bunch of things that you need to worry about, so kubernetes basically is designed so that upgrades can be relatively seamless, especially if you are using a managed offering where you can, maybe just like click a button and have it do some of those things for you. We've gone a long way since openstack when you had to like create a new cluster and then move everything over to that and then go and delete your old cluster and I've.

A

Seen like way too many people that are still running super old versions of kubernetes, because sometimes it's just easier to not do anything. um But please don't do that. um Even some managed offerings might have the option to do auto upgrades, so it just makes life a lot easier.

A

It's you're making sure that you have fewer of like things to do with the cves that have been found maybe years ago, that have been solved at this point in patched, basically try to stay up to date with um the newest and latest matches or as close to the newest release, as you possibly can, and if you're kind of scared about what could go wrong with updating it, probably won't be things blowing up when you're upgrading um it's more likely that you'll maybe run into something like a deprecation cycle, and then things are no longer working anymore so like how does that work?

A

So, for instance, say you have like some API version that needs to be deprecated so say that was like for Ingress for V1 beta1, so a new API was introduced, so there was a V1, and that was in 119.. So at that point you're, if you're using it you're going to get a bunch of warnings, saying hey this is being deprecated it'll be removed in 1.22..

A

You don't have to be like. Oh no, every the world's like ending I need to like worry about this immediately and everything else will things will blow up, however, like try to plan for it? Basically, once you start decommissioning your 118 cluster at this point, it's just migrate over at that point and remember: kubernetes will support both V1 and V1 beta 1 during that entire period. So maybe that's about like a year-ish or so so. Basically, you have a year to upgrade your yaml, so you don't really have an excuse there um for your enjoyment.

A

um This is the list of things that you have to do if you are upgrading a cluster yourself so and in the specific order that you have to do that in so, you have to update your control plane. You need to upgrade your nodes. You need to upgrade clients such as cube cuddle, and then you have to deal with things like adjusting manifests and other resources based on API changes that accompany the new kubernetes version.

A

So, as you can see, uh you really want to use managed offerings so that you don't have to deal with. All of that yourself, so quotas and limits they can be used. So noise neighbors, don't affect other workloads that are running on the same host.

A

They can also prevent attacker from using your infrastructure for using doing things like DDOS or crypto mining, which, unless you're trying to do the crypto mining, you probably don't want someone else doing that. You can also use things like taints and tolerations to schedule workloads away from each other. If you decide you want to, you can also add another layer of security by using sandbox container runtimes such as like G, visor, Arcata or firecracker.

A

Basically, they add another layer of compute isolation between the containerized workload and the underlying operating system kernel by adding another layer of virtualization.

A

By default on kubernetes, uh every single pod can talk to every single other pod. If you're not cool with that, then you need to use Network policies. Basically, kubernetes also assumes that you can trust the underlying Network. So, for example, the network of say, you're a cloud provider if you're using one, if you don't actually trust it, um one way to address that is to set up a service mesh with full end-to-end encryption, for example like mtls, some people might say that's Overkill, but all depends on what you specifically want.

A

You can also go the extra mile and you can use tools like psyllium for advanced Network policies such as applying like filtering on specific API routes. So, for example, you could have like some front-end client pod, and you want that to be able to talk to V1 users, but you don't want it to be able to talk to say V1 billing um using like ebpf magic, so Secrets um Secrets can have different rbac permissions. I will go a bit into what the heck our back is later.

A

If you don't already know, if you go hard car with that, you could even encrypt them at rest. Kubernetes supports that with encryption at rest, which will basically encrypt resources like such a secrets in etcd and basically that prevents parties that gain access to your NCD backups from being able to view the content of those secrets.

A

Also, please do not put your specific secret data inside of a commit, a config map, use actual secrets and then have your config map reference. Those.

A

So yeah, even better, uh don't put stuff directly in Secrets uh use something else that will put them in there for you. So, like the whole point, there is to avoid committing your secrets to say your git repo and then hey. Look the whole world sees it.

A

That kind of that's pretty bad I mean you may find that kind of obvious, but just as a reminder, um basically yeah, you don't want to manage Secrets yourself unless you really have to you, want to use something like hashicorp, vaults or sealed, Secrets or chasmus or Sops, or if you have are using a cloud provider, you can use KMS and basically yeah for a secret data. Encryption basically leverage the work that others have done to make your life easier and more secure, so people such as you all or your devs or robots.

A

So like say, if you have some automated services and you have code running on your clusters, all those things need to be able to talk to the kubernetes API so that they can deploy things.

A

um Whether so or maybe you have to scale up scale down roll out new versions do monitoring. Basically, all of the things, um but we don't want our Auto scaler to do things such as mining Bitcoin or like. Maybe maybe your developers ran away with like a credit card database, but probably more realistically, their laptops got stolen, which may have happened to me in on my last trip for kubecon.

A

um So then you just want to make sure that people can't access this data. So that's why this whole next section is uh for user management and permissions.

A

So we have good old separation between these two things, so you might if people might just say auth, but what kind of auth so there's auth n, which is authentication which is basically like who are you? Then there is also Aussie with his authorization, and that is okay. Well now we know who you are, but what are you allowed to do so for authentication?

A

um Kubernetes is a very flexible here. You could use things such as TLS certs, maybe like with your own root, CA or not uh IDC tokens with any oidc provider so like maybe it's like some in-house thing, like Dex or key cloak or SAS like OCTA, and that in turn can plug in with your clouds. I am if you're again using Cloud.

A

So basically that could map like cloud provider users to your kubernetes users, make sure to recall like what the best practices are so, for instance, prefer to use short-lived credentials so, for example, use oauth access tokens instead of like a username and a password so for humans, you might have to worry about like TLS IDC service accounts. Etc, like a fun note, is uh in kubernetes. Don't you don't just like create a user.

A

You give hey I, want to have the bill, this person or thing or whatever, to have the ability to create pods permissions for that to say like John, Deere or something like that, I don't know or Jane Doe or whatever, um or your name or some robot. But um basically what happens is as long as someone shows up with a valid cert or IDC token. For that user, then they can go and create pods and then robots deal with service accounts.

A

There is also the spiffy project which you can use for authenticating from one service to another. It's an advanced use case, but with it you can deal with things with like short-lived identities that can help you move away from these Long Live secrets, and if you want to look more into that, you can go to spiffy.io.

A

All right so authorization again, this is: do you have permissions to do whatever you are trying to do, whether it's like creating some pods leading something listing? Etc um kubernetes, the kubernetes API has uh several different authorization modes, so there's ABAC, which is attribute based Access Control.

A

There is node, which is a special purpose: authorization mode that grants permissions to cubelets, based on the pods that they're scheduled to run there's also web hook, authorization which allows you to use a custom author Z system, such as plugging in your existing system and the main one is rbac, which is what we'll look into next.

A

So, like the high level idea of kubernetes, is you go and you define a role which is a collection of permissions so like things that can be done so, for instance, being able to list pods, create certain types of resources, scaling up and down for some sort of resource, Etc and then there's being able to do things like.

A

Then you, you bind the role to a user or say whether it's a user or it could be a group or it could be a service account, and you use that with a role binding or a cluster role, binding also make sure to audit your rbac. There are a bunch of different Cube cuddle commands that you can use to audit permissions.

A

The first one that is listed there is there by default, to query the API authorization layer, there's other plugins that you can add, which is basically everything else there and just as a note, you can't take away permissions.

A

So basically, if you're like I, want to give someone full access, you can't be like oh well: I don't want them to have access to this specific namespace. You have to go and give specific access to each thing that you're trying to do. If you want to go about it that way, um permissions can be defined cluster wide or you can do it with a specific name space. So you can be like hey I want someone to only have access to this random sandbox namespace, but I don't want them to have access to prod.

A

So that can be a thing to like make do some separation there. You could, for instance, make someone even an admin in their own specific namespace, and if they blow everything up there, it was their namespace. So it won't destroy anything else.

A

So again do not give blanket admin permissions to just everyone that would basically be like giving root access, so namespacers namespaces basically make it easier to be lazy in doubt. Basically, if someone needs access to a bunch of stuff, create a namespace specifically for them, give them access to that and call it a day.

A

So software supply chain in 1982 there was the Chicago Tylenol murders. They can figure out like where along the supply chain, that there was something being put in Tylenol that was killing people. um So the solution was basically adding a seal there. So, for instance, transport is avoiding like a person in the middle attack. Your factory is your CI CD pipeline, so you can check it by having like a concept of a secure Factory.

A

So, for instance, you could have two CI CD pipelines and have them do the same, build you can check if the binaries are the same and then there's like thread modeling. It's like a definitive identifying like who the actors are in your organization, perhaps as disgruntled employees Etc. So if a new vulnerability gets disclosed, how do you a determine if you're affected and B? How do you deploy a fix or mitigation?

A

Are you like? Are you even sure that you aren't accidentally or someone is like unwillingly, running like a Bitcoin miner? That could be bad? um Basically, you can't be confident in the software that you're running unless you know where it came from and how it got there.

A

So there's a bunch of questions that you need to be asking or things that you need to be validating as you go from writing code to running it in production. So, like does a change come from a trusted person, one tool that you could use to help here is six stores get signed and you can use that sign commits so you're identifying Who, You Are and it's keyless, so you don't have to worry about gpg keys, so that's pretty cool. You can also use cosine for signing container images.

A

Next is the source code coming from your Source repo? Can only authorize people push to it when you build your artifact. Is that from a trusted source? Is the build system itself trusted? Can you trust like where it's running and what it's running on, and once your artifact or your program is built? Can you trust where that is being pushed to and that only trusted people can push to that as well?

A

And ideally, only your build system in your CI CD pipeline can do the pushing also make sure to do vulnerability scanning with your images to look for known cves. There are tools out there, such as Claire and trivi. Some container Registries such as Harbor, have this built in. You should also consider using an Opa implementation for policy enforcement, so gatekeeper is one of the options there. Sigster also provides an admission controller to assert image, signatures and other stations.

A

um There are some threat models and there's a bunch of different ways to mitigate them due to a time I might skip through this really fast but like, for instance, um so say someone manages to make a change to your code that you depend on and it's not your code, but something that you're pulling on. So someone decides hey I'm, going to delete this or it's accidentally deleted.

A

That could be a huge problem, so you might want to have immutable dependencies. Are you vendoring so that you don't have to worry about if it goes down? For instance, there's also things like attacks on uh like your build infra there was. This happened over with uh solarwinds and basically some code, that's produced by the build server, had malicious code injected in it and allowed external access So. To avoid things like this, you may want to have ephemeral, builds and to not have like a dedicated machine that someone can hack into.

A

So, basically, you would want like a build system that spins up on demand and then goes away afterward. Basically, if it doesn't exist, it can't be hacked um a few more things. um Long story short. If you spin up a cluster by clicking around in like some sort of web console or you use something like eks cuddle create cluster. The result is far from being production, ready, for instance, maybe backing up your cluster can be really important. You can use a tool such as Valero. Observability is another big thing.

A

Metrics can help you determine if something is going as expected, or if it's not and try to help you figure out what is happening.

A

Also good logging is indispensable for security, because we want to be able to audit who does what we want to be able to see things like user and privilege, changes and creation of things and security events, interruptions, logging, Etc and don't forget at the end of the day. This is running your code.

A

If you have problems with your code, kubernetes is not going to be able to catch them all, so you want to be able to use source code analysis tools such as oasp to help you analyze source code and or like compiled versions of code to find security flaws here are some other resources, some of which I guess. I have already mentioned.

A

I'm doing this talk again, hopefully with less technical difficulties at Spring, one in December, and there will be other talks both on like kubernetes and spring, if you've never heard of spring. It's because you're, probably not in the Java world, but there's that so if you want to come over there, um if you could fill out the feedback, that would be super helpful. That way, I can work on improving this talk, so that would be great I'll. Leave that up there for a second, since I apparently have like two minutes.

A

All right well, thank you very much um come up and talk to me. Afterward I think I'm pretty much at the end of this year, but yeah so special thanks to a bunch of folks who helped me learn some of this, because uh I went into this with I. Don't know anything about security. I should learn it. So therefore, I'm going to propose a talk, that's about security and force myself to do it. So that's how this went so yeah again, there's my Twitter and thanks again, everyone.