Cloud Native Computing Foundation Online Programs, 5 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing Kubernetes Container Platforms in 2021

Description

Using the Kubernetes Security Guidance best practices published by NSA and CISA in August of 2021, Lars guides us through the field of Kubernetes and security. The whole range of security tools at our disposal is covered, and we will learn how to secure the Kubernetes control plane, ensure that containers and Pods are deployed in a security-minded way, and what to do to prevent attacks. Is it sufficient to just run Kubescape, to follow the guidelines, or what else do we need to do to get a secure Kubernetes-based container platform in 2021?

Lars has worked with cloud technology since 2008, both within industry and academia. He holds a PhD in computer science and is a senior cloud architect at Elastisys. He is one of the architects of Compliant Kubernetes, a CNCF certified Kubernetes distribution, designed and developed especially to meet the high security demands of regulated industries.

The blog posts with additional information are here:

https://elastisys.com/nsa-and-cisa-kubernetes-security-guidance-summarized-and-explained/

https://containerjournal.com/features/hardening-kubernetes-beyond-nsa-cisa-guidance/

Check out Compliant Kubernetes, the Kubernetes distribution for regulated industries that Lars talked about, at https://compliantkubernetes.io/

Connect with Lars on LinkedIn at: https://www.linkedin.com/in/llarsson/

Follow Elastisys on LinkedIn at: https://www.linkedin.com/company/elastisys

Connect/follow me:
https://www.linkedin.com/in/joebignell/

A

So if everybody is ready, we are going to start, so I will start off by making a very quick introduction of myself. So my name is joe bignall. I am the host of devops for everyone. I see some familiar names in here, which is fantastic. I see some new names, which is equally as good as well devops for everyone has been around since march 2020 when we did the first ever meetup in person in central london.

A

um Since then a little thing called covered happened and we've gone online, which has actually worked in our favor, because people like lars has been on the group now, for, I think, maybe the second or third time so lars and elastasis have become a fantastic partner to devops for everyone so pleased his back today because he's very knowledgeable guy, a very, very knowledgeable group of people that work at elasticist and, as you can see on the screen here, we're going to be hearing about lars's, summarization and explanation on.

A

What's in the us, nsa and cisa kubernetes hardening guidelines publication. So lars is an employee at elasticis. In sweden, one of the senior cloud architects and branch manager, so somebody that has a wealth of knowledge in this field. So we are very, very happy that he's here today and for those that don't know elastasis, it's a swedish-born company that provides products and services for scalable and responsive it systems, I think, lies if I'm not mistaken, an emphasis on predictive, auto scaling and multi-cloud capabilities.

A

So a business like I say, that's been a great partner to devops for everyone and really looking forward to this chat today.

B

Thank you, yeah. Thank you joan. Thank you for having me back again. I think it's wonderful to be back again. It's awesome. I I just love it and yeah. It was funny that you said that you started in march of 2020 because I was like hmm and then what happened. Yeah.

A

C

B

So so this this uh to me, this has been a very virtual meetup, but I would love to actually be there in person once that is possible again. So um this is me and, as joe said, I've been doing this for quite a while. I've been doing cloud computing since 2008, or so both in industry and in academia. So I have a phd in computer science for stuff that I did in cloud resource management, but my heart has always been you know in sort of the terminal and linux systems and stuff like that.

B

So I have tons of experience in software engineering and devops. um I work as a senior cloud architect at leicester's and we make this open source kubernetes distribution called compliant kubernetes, which is a kubernetes distribution that is tailor-made specifically for organizations that are in regulated businesses. So you know education or medical.

B

The medical field finance that type of just general industry, where there are tons of regulations that you need to follow, and they have very strict requirements on information, security and uh I'd love for you to follow me on linkedin joe we're making these slides available right, I mean you can like download these uh somewhere in the future right.

A

Absolutely yep so there'll be something to be able to download and then reference back.

B

Okay, awesome yeah, because then you can actually click on the things that are linked in this. So.

A

Makes sense yeah, that's good.

B

Yeah right, so the outline for uh the talk today is that I will first briefly just mention the different types of threats that are in this nsa and cisa report, so kubernetes hardening guidance that they released in august this year.

B

First of all, I will summarize and explain it, but then I will also go beyond it, because I think that while they cover a lot of the basics, I also feel they don't address everything, at least in the way that I would have so because I've been doing this stuff for a while.

B

I have some additional perspective and I hope that you will uh both enjoy listening to it and that you will agree that uh it makes sense to go beyond what they propose and, as joe said, if you do have a question, then please just like unmute yourself. Is that even possible joe yeah, because you no?

B

Can someone try to unmute themselves and just say hi.

C

B

Hello awesome, so it worked perfect, perfect. Thank you for helping us figure out these technological things. So.

D

We know they're working you'll, see the chat way in the way if you want to there's some inconvenient controls on top of your screen. If you hover with your mouse in the middle of the top of the screen.

B

Oh okay, ooh, yes yeah! That was yes that was inconvenient.

B

Perfect, okay, thank you! That's awesome, yeah! So I'll keep that up as well. No, I can't actually because it's going to cover my presentation for me I mean I, I know what it's about, but I still want to see it. Okay. So, let's start with a really uncomfortable question: okay, so the uncomfortable question is what permissions have you given me in your kubernetes cluster or cloud infrastructure?

B

If I manage to hack into your application now, I know if this was a real event and I could like see your faces.

B

I would have like two or three like knowing smirks, you know shining in my face, but most people would sit there with either uh sort of light, panic or just a blank stare, and what we will talk about during this presentation will hopefully inform those with the blank stares and with the panic in their faces, so that you in the future can have more of that knowing smirk like yeah yeah, I know exactly the answers to this one um and that it's actually a a nice one, we'll get we'll get back to what the good answer should be to this one.

B

So what are the threats? Well, there are three types that the report covers and I think it's a very good distinction between the three. So first of all supply chain risks. Now hardwiring software comes from somewhere else. Most of it, you write your application, but that's just a tiny, tiny, tiny part of all the software that is running in your system. So do you actually verify that it is all correct, probably not unless you're like in the defense industry?

B

You're, probably not that paranoid, but if you do deal with really sensitive information or possibly classified information, you should be aware that this you can actually be targeted by hardware, vendors and software vendors as well, especially, you can't just like pull any random docker container off of docker hub and hope that it just like works, because maybe there's stuff in there that you didn't like we'll get back to some of that as well.

B

Of course, there are the malicious threat actors, you know hackers outside threats uh and we also have insider threats. Now these are like internal users that they actually do have legitimate permissions to access your systems.

B

The problem is, of course, also that two-thirds of all insider incidents are due to negligence, so this is that they could either be intentionally malicious. They could be coerced by outsiders, but they could also just be clumsy, and two-thirds of all insider incidents are apparently due to negligence, which is you know fast word, I'm guessing for clumsy, plus that there are policies that don't really protect the system there. There aren't any policies in place that are being enforced. That would protect the systems right.

B

So, let's, first of all go to what the report actually covers.

B

How many have actually seen this report by the way? Oh yeah, I can't do show of patents, damn it hopefully you've seen this report. If not, when you have the slides, you can click on that link and what it is. It's a 59-page tech report and it was published out in the public in august this year it's focused on kubernetes itself, so there is a very detailed discussion of security related configuration of kubernetes.

B

So how should you configure your api server in kubernetes? How should you make sure to restrict access to the control plane? Part of kubernetes, etc, etc. There's also a whole lot of example code. So it is both.

B

Rather high level, when you just read the executive summary, but it's also very like low level in the sense that it talks about, like which network ports do you need to open and why so, it mixes both those perspectives, and I hope that this presentation is somewhere in between, because what I will do is that I will take the executive summary points and I will expand on them to make it more understandable like why. What are you actually achieving when you do this, because it's a government report, it doesn't really say anything about.

B

You should use this or this or this software, because I'm guessing that would be uh wrong and yeah. So it only mentions other security software in passing my main problem with it, though, is that just like I see you know after reality. Far too often, it focuses mainly on like hardening as sort of a one-time thing that you do rather than uh the process that I think that security, actually is you know, so it's it's a continuous process that you can't just say: yep we did security and then you're done.

B

You know it's not like a check box that you just like yeah. We did it. No it's a process because threats are evolving, and so must your security posture.

B

So the first thing that they write about is that you need to scan containers and pawns for vulnerabilities or misconfigurations and the problem or one of the problems with containers and security. I'm sure you're, aware that container images are immutable.

B

That is, they don't change, which is wonderful, because that's why we love containers, I mean we want them to not change, because that means that we can just let you know here is the container image that we tested in our qa environment and then we can promote that to staging and we can promote the same one to production, and we know that there are no underlying changes. It is the same.

B

However, that also means that you have a whole lot of insecure code that will stay there in a time capsule, because if you don't update the software, but threats uh get updated out there in reality, you will actually have code. That is running. That is not secure, and the ironic thing is that the more stable a component is in your deployed system, the more infrequent you update it, the worse that actually is for security.

B

So there are vulnerability scanners. You can actually scan containers using various pieces of software.

B

I recommend one called trivi, but basically all docker registries have some kind of scanning functionality in them right now and what they will do is that they will scan when you push the image, but the problem- and the report says also to don't just scan when you push scan using an admission controller now in kubernetes, an admission controller is a component that catches sort of the api call that says you know please run this container for me, run this as a pod and okay. That is great.

B

But if I have this component in my in my software service, that is only updated say once a month or less often than that. Even that means that when I push it to the registry, well, maybe it doesn't have any known vulnerabilities.

B

Then, when I deploy it, that's probably like a couple of seconds after I pushed it, so that doesn't really matter so what I think you should do and I'll get back to this as well is, I think you should run a daily scan of all images that are that are in active use, so just figure out all the container images that you have deployed and scan those, because you need to be on top of any new uh vulnerabilities that come out.

B

Number two run containers and pods with the least privileges possible.

B

So, as you may also be aware, with containers and security, basically, the default settings for containers and also kubernetes has been to not really care all that much about security, so containers run as the root user by default, and that's not good at all, and the report also says you know if you have this container and it has a file system, but it's not actually supposed to write anything to the file system. Why would you even allow it so use read-only file systems until need arises and in kubernetes there's also this?

B

There have also been these security features. You can actually set the so-called security context and you can use pod security policies. If you have. You know previous versions of kubernetes or pod security standard, if you have the version 1.22 and later so these are essentially settings that are the best practice settings for how to run a pod in a secure way, which means that it can't do various things against, say.

B

The host operating system there's also a problem in that when kubernetes runs a pod, it actually gives permissions to that pod to work with the kubernetes api. Very seldom is not actually something that your software needs to do, but it just gets the permission to do it.

B

Why so my take is that you should do all of that that the report suggests, of course, but then you should also encode your policies for automatic enforcement via open policy agent, and there can be many other policies and I'll get to those as well, because there is just so much more fine-grained policy enforcement that you can do and that I, that I think that you should do as well.

B

The report says you also need to use network separation to control the amount of damage that a compromise can cause. Now this is key because the default settings in kubernetes they allow all plants to network with all other pawns, regardless of which namespace they're in regardless of whether they it really makes sense to or not, and the problem is, of course, that the security of your entire deployment is only as strong as its weakest point.

B

So if I manage to get into you know some component that you basically it's a throwaway component, that you don't think that anything is going to happen to it. Well, that can then serve as the bridge into the rest of your system and what you can do and what you should do is you should enable something called network policies, and these are kubernetes aware, firewall rules, so you can then either specify rules that say you know you. This component can talk to these ip blocks, or these components can network with these other components in kubernetes.

B

So then, you can say a rule that basically says only allow components that are marked backend, to connect to components, marked database and nothing else, so nothing else can actually connect to database. Also, backend can't connect to anything else. So then, you can really limit the amount of exposure that you have if anyone would manage to hack into your system.

B

But that's like within the cluster, so they also recommend using firewalls to limit unneeded network connectivity uh like on the cluster node level and also to kubernetes core components.

B

So if you have this kubernetes cluster and it has an api server and you should only be able to access that from within local area network or so then, of course you shouldn't expose it to the public internet and of course you should also only restrict it so that only trusted users could possibly access it, because why make it easy for automated vulnerability scanning tools to find like hey, here's, a vulnerable, kubernetes api server, ah I'm gonna, I'm gonna launch an automated attack against it.

B

That's that doesn't make any sense, so the same goes also for the uh etcd database in kubernetes, because that's sort of the brain of kubernetes, together with the controller managers, so restrict access to all those core components and then also it doesn't really make sense to have unencrypted by default network traffic in kubernetes clusters.

B

If you are running on untrusted networks and I'm not sure what that means for your organization, so to some organizations, um the cloud would be an untrusted network so like in in amazon. You wouldn't trust that for some organizations some organizations make a different risk analysis and they say no, no amazon is fine, but you know if we ever need to run like on-premise at a customer location that is untrusted.

B

So what I think you should do is you should use a networking provider with transparent encryption in it. So wireguard is this new type of vpn that you can actually use as a feature in calco and there you just have automatic encryption. You don't need to worry about it and the pods aren't even aware. So you just get automatic encryption for all network traffic. That flows between hosts in your kubernetes cluster, perfect.

B

So kubernetes ships with support for authorization, it doesn't have any smart system for authentication. So access control is a problem in kubernetes clusters.

B

There is support for understanding, openid connect and a few other options, but you need to actually opt into them, so you need to actually actively choose to have authentication features.

B

So authentication would be the thing where I somehow prove that I am me so some system says this is lars. He works at elasticist. He is a member of these groups and that's it that's where authentication stops and then authorization is where you assign roles to users. So there are also features for that included in kubernetes, but they aren't enabled by default.

B

So you need to actually enable those and you need to configure them, because, even though you enable them, they don't have any rules in them. So essentially you need to teach kubernetes how to access.

B

But if you have open id connect integration and you have role-based access control enabled, then you actually can have these rules in place. So what I think you should do is you should definitely disable this perpetual admin token that you get when you create a kubernetes cluster, because that's that's perpetual, it's never going to expire. You always have like a way in using that, and you know that's that's a big problem, because even if you like enable audit logs, it's not gonna show up as anything it's not gonna show up as a specific user.

B

So if you have that token, you are basically hidden from all kinds of nice auditing, but you can do everything you should enable openid connect for user and group membership so that you can integrate with say, azure or google or whatever you can. You can do that via dex, for instance, or key cloak. uh We ship dext in compliant kubernetes and we're happy with that, and I mean if, if that suits your needs, that's a great way to connect to various identity providers and spit out openid connect tokens that kubernetes can then use anonymous.

B

Access is also enabled by default in kubernetes for some reason, so that should be disabled. You should enable role-based access control and you should also have a whole bunch of rules in place to restrict permissions as much as possible when you have role-based access control in place.

B

Now I mentioned audit logging, so all api calls can be logged for auditing purposes, but there are a lot of audit. There are a lot of api calls that will wind up in your audit logs, because every time, a cloud, a controller in kubernetes does anything it's interacting with the kubernetes api and all of those calls will also wind up in your audit logs.

B

So while it's great to collect this information, you also need something to help make sense of it. So the report says you should use some kind of automated system, and my take is that there's this program by sysdig, it's open source, it's called falco and it can act both as a simple security incident and event management system, and you should, if you do that, together with say, a centralized logging system such as elasticsearch, but I'm guessing you're running some kind of logging solution. Anyways.

B

The report also says that you should periodically review all kubernetes settings and use some kind of vulnerability scans to help ensure that risks are appropriately accounted for and that security patches are applied, and that is something that I'm guessing that everyone here in the audience knows exactly what a big pain in the neck. That is because kubernetes releases new versions.

B

Quite often, there are three uh releases per year or so, if you check the history and kubernetes supports officially current version and two uh and the two before it so n, minus three is what they call it, and that means that if it's three times per year at least you have to update kubernetes once a year, but I wouldn't recommend doing only every every third version because they add a whole lot of security related features as well.

B

The problem is, of course, as this also says that you need to review kubernetes settings. Is that all kinds of security features, as as you've noticed by now, is that they are typically opt-in rather than opt out, and you definitely need to opt in as soon as possible, and there are tools out there that can run some automated tests and they can help find insecure settings, and the problem is, of course, that the insecure ones are the default ones.

B

So I think you should definitely run automated testing you need to, but are they sufficient? So there are two uh that are like top of my mind. Cubescape by armo security is a really nice tool. They have released a framework, basically a set of rules that checks against exactly the well some of these ones that are in the report.

B

However, it currently relies on what it can determine via kubernetes api calls. Cuba bench by aqua security connects deeply into the control plane host node, like one of your, you know previously called master nodes and therefore it can currently inspect more than cubescape can, but both of them really help. You understand the low hanging fruits of vulnerability scanning, so I think you definitely need to do this, because obviously anyone trying to attack your kubernetes cluster will also do it and not having done this.

B

Basically just screams, there's an insecure cluster over here and if they find that, then you are in real trouble now. The problem with both of these and with all really is that they are limited in what they actually can investigate. So they can't verify that, for instance, your storage provider does encryption at rest. They can't inspect any firewall rules or security policies encoded in other systems than kubernetes.

B

They don't know anything about the underlying operating system or any third-party software or third-party services that you use, and they will never be able to tell you that your policies are actually sufficient for your information security needs, because how would a piece of software know that? So it can't actually, but they can definitely tell you if your cluster is going to be fine for automated tests or if there are any obvious problems, and you don't want to showcase any obvious problems ever so. That's basically, where the report stops and where my suggestions begin.

B

First of all, I think you need to be very strict with preventing misconfiguration and not just checking for it. After the fact, as I said before, if two-thirds of all insider incidents are due to negligence, that means that there weren't sufficient controls in place to ensure that policies were adhered to and like role-based access control is great, but it's also limited in what it can actually express. So I mean a role-based access.

B

Control system can tell you like lars is allowed to modify configuration in the production environment, but it doesn't say anything about like what changes am I actually allowed to make? Can I set stupid default passwords? Can I make the password of the database like one two, three, four five, I shouldn't I'm guessing be allowed to do that, but rollback's role based access control.

B

All that can do is, it can say, you know, is lars allowed to do it and to that yep thumbs up so open policy agent can actually look at all incoming api requests to the kubernetes api server, which includes you know, changes in configuration, and it can have this large set of rules that it checks against and either gives a thumbs up, which means that the request will be processed by the api server or it gives a thumbs down and the request stops then there and is never processed further by the api server and it's a bit weird language to use.

B

To be honest, it's this logic based language which, if you are a prologue programmer, you're comfortable with that. If you're not, you can look in the library of the rules that already exist, or you can check for third party rules as well to have some inspiration.

B

The problem is- and I think everybody is probably like technically aware of this, but they don't think about it like day in and day out, and that is that any permission that you give to an application will also be given to uh bad actors. uh Malicious users, hackers, whatever so a hacked application, will have all the permissions that the application usually has so now, all of a sudden that hacked application can go in to your third party software as a service integrations.

B

If you have this vpn or something that connects back to your back office, you know the hacker can use that too. It can access databases freely.

B

Probablyo can also quite easily determine how to connect to the database, so you always need to restrict your app components as much as possible, because- and this is something that I see like all the time like- why would something that is just supposed to like get a rest? You know incoming request over some rest api. Why would it ever get to do anything besides get the request and do some processing and send back the results it shouldn't write anything to uh like local storage.

B

It shouldn't do any integration with third party applications unless it absolutely needs to so always restrict as much as possible.

B

You also need to keep cloud resources specifically in mind. So I mean say: amazon has like what 200 something services and it's great. They have great services, but it can also be quite cumbersome to use them, so it's very tempting to install like various controllers and operators that are out there in the cloud native ecosystem and community, and they can offer like various cloud integrations like.

B

Oh, I can take care of this stuff for you, but you really need to check the code and also documentation, of course like how seriously do they actually take cloud security and you need to reject the ones that aren't clear about how configurable they are with regard to what permissions do they set on these cloud resources that they manage for you and also, you should, of course restrict as much as possible for them as well.

B

So I will never install anything unless it also specifically says what permissions will it set and what permissions does it actually need to run? Did anyone have a question? I thought I heard somebody say something: no, if you do have questions, then please interrupt.

B

Here's a question: does your application unintentionally have permissions in your cloud because that's a big problem, so what I've seen many many many times is that uh people assign you know these instant instance profiles that basically give worker nodes in your cloud the ability to modify, say, dns records or auto scaling groups. You know so you can scale the cluster up and down or whatever the ability to modify load balancers, because the problem is, if those apply to the server itself, if they apply to like the worker node.

B

That means it applies to any application running on there as well, and if I manage to hack into that application, then all of a sudden I can act as your virtual machine in the cloud and then I have those permissions as well because to the cloud.

B

It is a call that comes from that server and all you need to do as a hacker is you can just figure out like hi cloud? Can I have you know my permissions now and then please run these api calls for me. So all of a sudden, maybe you can scale up the amount of servers that are running and then you can. I don't know mine, bitcoins or whatever you want to do with a whole bunch of extra free capacity.

B

As I said before, you need to regularly scan all your deployed container images, not just when they are new and like have been pushed to the to the image registry. So what I think you should do you should loop through all your pawns that are currently deployed. You should determine which container images are in active use in those pawns, and you should scan those images, and if you do that daily, then you have the best possible.

B

You know support from this vulnerability scanning tool that you are using. So I think that is much more secure than the model scan on push or scan on deployment that the nsa and cesar report said.

B

This one is the one that I love the most because it's so much fun for your engineers to to break stuff, because basically building is hard, but breaking is easy and it's also fun like if you have kids you've seen this, you know, building lego or whatever that's hard or building with dominoes that's hard, but breaking is really easy and it's way more fun.

B

I think you should regularly have your own staff security test your entire system and make it a half day or full day event, because it's it's a lot of fun and you will see a whole lot of creativity coming out of your staff, so your engineers have access to source code, which hackers don't and if you let your engineers break your application.

B

It's way better if they find an errors than if hackers do what this will do is it will foster this security first mindset, and that is key because often security. Is that thing that you know you you?

B

I don't know you have a security review every once in a while, and then you just get issues that wind up on some backlog and then they get ignored forever and it doesn't do anyone any good, so they just I don't know you take care of like the top five most critical ones, the one that seems like high priority and you never take care of the rest of them.

B

That's not a whole lot of fun. You need to have this security first mindset instead, and this is a way to communicate that you actually take this stuff seriously. um I've done this.

B

I've done this for customers, and you know you really should also keep stuff like denial of service in mind when you do this because for one system for instance, I could break that very easily, but with just my laptop, I just made normal requests, but just far more of a certain type than the engineers had ever thought was reasonable, um like I created more groups and nested groups than was ever intended.

B

But of course I could do it because the system allowed me to, but it also made the database really slow for everybody, and that was an interesting bug to find so do this. You'll have a lot of fun and you'll be able to catch errors before the hackers do, and that is so much better.

B

Disaster recovery, another thing that you're supposed to have a plan for.

B

Let's also point out that disaster recovery is not equal to backups. So what is a good disaster recovery plan? Well, you need to so many people understand that okay, so a disaster could be an entire cloud region goes out, but do they have a plan and do they actually practice for dealing with that type of situation?

B

What that situation would be like is that, basically, you can't trust your infrastructure anymore, and you need to recreate it.

B

I think you should also practice this from a security point of view, because, basically, is it possible for you to see this large-scale attack as a disaster, and then maybe, if you could practice like okay, so we need to nuke everything we have. We need to recreate everything in you. We need to restore data to what it was like like five hours ago before this attack started and, of course, we also need to you know plug whatever hole the hackers got in through.

B

So you should really have a disaster recovery plan in place, and I know I know from experience that basically everybody knows they should have this, but they don't have it in place. But please prioritize it because large-scale hacking is a disaster to your business.

B

There are many nice systems that you can also use so intrusion detection systems. They will tell you about ongoing attacks, so basically, you determine somehow either via rules that you encode or via some kind of machine learning thing how the normal operation of an application yeah.

B

What that looks like so what's the normal behavior and then an intrusion detection system can tell you when the application starts to deviate now what if an application that never reads or writes outside of a certain directory, all of a sudden tries to read and write files outside of that directory.

B

It's not like the code just started to make that up.

B

It's probably somebody that is trying to exploit something in your application, so an intrusion detection system should be able to pick that up and understand that hmm you know the application that usually just writes here is starting to write over there and that's a problem, and then it should warn you and that also feeds into this other automated system, uh security, information and event, management, which searches through all kinds of logging systems, like both audit logs that we talked about before, but also uh logs from applications, and they will try to find and flag abnormal events.

B

Now there can be false positives, but it could also be serious indications of incidents, and luckily, there's this open source thing again called falco which acts both as an ids and also this cm system.

B

So, in summary, I think that the main message here is that kubernetes is neither safe by default, nor by itself you really need to make sure that you restrict access, in both terms of network, to users, to machines, and also you need to restrict privileges.

B

You need to periodically use tools to assess your current security practices. You should prevent misconfiguration, not just check after the fact you need to take running in the cloud and any permissions also in the cloud into consideration. So please be mindful of that.

B

You definitely benefit from a security, conscious, engineering culture and please take the disaster recovery problem seriously, so apply also disaster recovery plans and how to practice and how to actually uh run through these plans. I apply that also to security breaches.

B

So going back to that uncomfortable question from the start, you know what permissions have you given me in your kubernetes cluster or cloud infrastructure? If I manage to hack into your application, hopefully the answer should be no more than absolutely needed.

B

You know for normal operation of the application and that you will see that I'm there, because you have automated systems that both limit what I can do and they also raise an alert. When I make the application behave in ways it doesn't normally do so that you will catch me. You will see that I'm there and you won't be fooled.

B

Okay, are there any questions so either you can ask now and now I'm gonna try to bring up this. This um chat and if you have questions that you haven't typed into this chat, then you can send them via email. If you want, you can find me on linkedin, you can, you know, connect with me there. You can send some questions there.

B

um Are there any questions? Oh yeah, I'm scrolling up and somebody says, check out m9sweeper.io a freemium scanner for kubernetes does ongoing scans in the cluster, as you suggested, takes less than 10 minutes to install was demonstrated this summer on this podcast awesome, yeah, perfect, that's so nice, so m9sweeper.io, I'm going to check out uh that system. I hope everybody else does as well.

B

Any live questions.

B

How about using caverno instead of oppa yeah sure so they solve the same general problem, because they both act as this admission controller again. So the thing that basically intercepts api calls coming into the kubernetes api server and it, and they both have this set of rules. They can say thumbs up or thumbs down to that api request and basically caverno and oppa. They have different. um There's the language.

B

How you specify rules is different and sure I happen to like oppa and gatekeeper, even though it has a bit of a weird language to use, simply because it's it's something that's been around, you can integrate it into other pieces of software as well, because you can use it as a library, and I've had some great success with that. But yes do check out caverno as well.

B

If that is nicer and fits your mental model of how these things should work and how you want to write rules, then by all means use that just use something any other questions by the way. Thank you for the question about caverno.

B

Ali says I have tried to implement pods to run as minimum permissions, but sometimes I miss them when deploying public health charts. Are there tools that you know that will tell the state of the cluster which containers are running with root or higher security context?

B

So what I think you can do in that case. So, basically, the question is: can you scan after the fact if containers are running with poor security defaults? And yes, I believe that is part of cubescape, so try cubescape, because I think it's going to tell you, but what I think you should do again.

B

This goes back to I mean I'm starting to sound. Like this broken record, you should have open policy agent in place. I think, because that will look at all incoming api requests, even if they are made by helm, and it will make sure that you can't break policy.

B

You will heardlund with a whole lot of uh acronyms after his name asks.

D

C

B

Would you use kubernetes for processing large-scale gdpr article 9 style, sensitive personal data? If so, what would be your approach? uh Would you be willing to enlighten people about gdpr, article 9 style, sensitive personal data.

D

Yeah, that's special categories of personal data, like data about an individual's health or sexuality or religion or persecution, stuff stuff, you can blackmail people over. Basically, if it leaks, then there may be a national disaster.

B

Yes, all right, I would and the reason why I say that I would is, of course, that that is the whole point of that compliant kubernetes distribution, that I was talking about that we make. So if you process that type of information, you need to be very mindful of what cloud provider, if any that you use or if you maybe run an on-premise kubernetes cluster. So I mean your security that the information security.

B

Isn't really affected in the sense that I mean you need to affect it. You don't need to uh like. Oh I'm running I'm running in the cloud. That means I'm sending it like to say amazon. No, it doesn't need to be. I mean you can actually benefit from kubernetes, even if you run on premise, because I'm guessing that you want to process this type of information say if it's about say, if it's about you know, patient data.

D

Maybe you need.

B

Yeah, maybe you need to uh process this on premise, maybe even like in the hospital's own basement. Essentially they might have a data center, so.

D

Good at guessing.

B

Yeah right so maybe that's what they require and then you can use kubernetes there, because the point is, you want to use kubernetes because it helps accelerate your developers. You want to be able to push out software. You want to use cloud native uh tools you want to use, like I don't know, like postgres, managed by say the uh zorlando postgres operator.

B

I don't know why I mean there are many reasons why you would like to use kubernetes, but that doesn't mean no use of kubernetes does not imply we're sending it all to this public cloud. That's run in the us or whatever it's not the same. It's not it's not neces, it's not a necessary step. So uh in compliance kubernetes we use this tool called cubespray to install and that can work on bare metal, anything that basically has an ip address and runs linux.

B

You can install kubernetes onto it and you can then take it from there. So I would definitely use kubernetes for processing uh that type of information, no problem.

D

B

Yeah right, anyone else have any questions.

B

Yeah, so I didn't mean for this to turn into the sort of compliant kubernetes sales pitch, because I don't think that's what this meetup is supposed to be, but go check out the open source projects documentation so go to compliant kubernetes.io.

B

If you want to learn more about that project, we do also manage it as a service on top of various, both the u.s, but also eu-based cloud providers, specifically for the reasons that we talked about during this last question, all right so joe any closing words.

A

Yeah, absolutely, I appreciate all the information and answering all the questions as well, so thanks for the uh interactiveness from everybody in the audience lars, I believe there is also a blog post that covers a lot of what has been mentioned today. Is that right.

B

Oh yeah, there are actually two now the second one is actually public as well. We sent it for uh publication with kubecon, and now it's also public, which is kind of strange, because cubecon is like what in a couple of weeks. uh But yes, so uh there is one blog post uh where I've summarized uh the nsa and sisa guidance report there. You know the kubernetes security uh hardening guidance, so that's on elasticist.com.

B

You can find the blog there and the other one is. I think in containerjournal.com.

B

Yeah thanks awesome.

B

Can you can you also link to the uh yeah to our own blog post? Maybe.

A

She's on the ball.

B

Awesome uh I uh yeah, so I also got a uh question sent privately to me. So I'm not gonna read that person's name um is essentially. The question is: do you need to have a person that is an expert in compliant kubernetes uh before you try to implement compliance kubernetes in a large enterprise or what's the story? Basically? And uh no uh you don't need to. I mean it's compliant kubernetes. uh If you go to compliant kubernetes dot, io you'll see that basically we install using cube spray, which is this standard tool for installing kubernetes.

B

We have our own fork, so we can implement changes faster, but then we upstream any changes that are of relevant use for everybody else.

B

We've added, you know: support for vsphere, uh because on-premise installations uh and for cloud providers such as exoscale and up cloud and other other cloud providers, so using cubespray, you install kubernetes, and then you put our applications uh on top of that, and that gives you this fully featured cluster, which you know has all kinds of nice things like backups and uh monitoring tools and log processing, and you know, access control, all kinds of nice things that you would probably anyway want to build yourself yeah.

B

So no, you don't need a an expert from us. You can figure that stuff out on your own, but if you would rather have somebody from us come and help you, then obviously we can have it's all open source and you can find it all documented on compliant kubernetes dot, io. Anyone else.

B

If you do have any questions, I think time is running out, but if you do have any questions, then please by all means do send them either via email or find me on linkedin.

B

Thank you so much and thank you joe for having me here again.

A

Not a problem absolute pleasure. Thank you very much and thanks for your time and thanks everybody for staying tuned. um Lars is a great guy to fire questions out. So his linkedin and his email address is on the screen, so definitely use last one of the kubernetes questions, but until next time lars have a great day and I'll catch up with you guys soon.

B

Wonderful, thank you bye. Everyone all the best.

A

Everyone stay safe, bye.

A