Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hack Back; Let’s Learn Security With CTFs! - Lewis Denham-Parry, Chainguard & Natalia Reka Ivanko

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Hack Back; Let’s Learn Security With CTFs! - Lewis Denham-Parry, Chainguard & Natalia Reka Ivanko, Isovalent

Speakers: Lewis Denham-Parry, Natalia Reka Ivanko
Threat actors have always been looking to attack clusters. Do you have the right security in place to detect and defeat if they are targeting yours? Or they are already in? Kubernetes has become the de facto cloud operating system and production environments have increased in maturity. So have the threats. Security Teams don’t necessarily have the expertise to detect state-of-art attack scenarios specific to cloud-native environments, like Kubernetes. So, where do they get started? Capture-The-Flag (CTF) events are a great way to learn about the techniques of both attack (Red Team) and defence (Blue Team). This talk will give you a framework for your own internal CTF events, with Red and Blue Team assessments, as a best practice for improving security in your organisation. We'll give a hands-on, live walkthrough of the top 3 state-of-art attack scenarios as CTF exercises using common open source projects like Simulator and Tetragon. Remember, the best way to learn how to detect is to first know how to attack!

A

Got the stage I like just to raise awareness to mental health um I've suffer from depression in the past and still do today. uh It sucks and I know that there's lots of mental health issues going on. We have conferences like we're at it's great to see people again, but equally, there's lots of fear of missing out is I need to be somewhere. So um just make sure you take a break and.

B

A

There's always someone who wants to if you ever feel rough or tough, then there's always someone who's going to listen and reach out to, but there's something else. um I just wanted to do a quick bash warm-up because who doesn't want to do a bash warm-up um whenever I run this command, um something happens which I don't expect it to so, if false, so to me, false is a negative and so I'm expecting no to come out.

A

But when I run this statement it keeps returning yes and the reason is, is because I didn't appreciate the boundaries that were in place around that statement. um This year, I've had a number of friends. Who've had people come up to them and even though they've said no, uh it seems like the other person heard. Yes, I'm only raising this because I've seen it really affect people and it sucks and um yeah I. Don't I, don't know what this will do for it, but yeah.

C

All right so who are we I'm Natalia I'm, a security product owner at isobail and I, really like surfing and running and like flying and generally being outside, but I, also like reading good books and listening to good music? And here is Lewis.

A

Hi I'm Lewis I'm, a security architect at chain guard I'm, very impressionable, as you can see with my list of emojis um and that's pretty much me.

C

And we are here to give you like kind of a couple of tools and kind of like an architecture um to run CTF exercises inside your company or just like with your friends and basically to have fun and then I work for eyes available and Luis is working for chain guard. But this stock is not specific about companies or like running specific tools. It's basically giving you like kind of like a CTF framework and an architecture. But if you are at cubecon, we are here as well. So please come to say hi to our booth.

C

And then, if you are here virtually please come to say hello.

A

Has anyone seen this comic strike before from outsk uh XKCD?

A

So for me, this is the problem that we have at the moment with ourselves in that we learned something new, but we're too we're too afraid to be able to share that on to someone else, because we're afraid that if we open up to say oh, have you seen this new thing and someone says back to you: God yeah I knew about that already. How come you didn't know about that? It's something I feel that we need to overcome. We need to be more open, and this demonstrates it exactly how I think of it.

A

I I had the role of a trainer for a while and just helping people get started the first time they popped their first shelf out of a container. It's that magical moment where it's like, oh God, actually I can do this again and again, I'm being able to share with that with people is just it's Epic.

A

So what is a CTF, so it's somewhere between a company offsite and an escape room and the movie Saw full.

A

So um if you go into the escape room and someone's like well I'll get out of here by throwing a chair of a window, yeah yeah, you congratulations, you've done it, um but we probably need to respect our boundaries unequally. We need to put ctfs in place that are achievable, because if the CTF has a keyboard, that's about a foot away from my hand and I'm chained back and there's a hacksaw. Next to my hand, that's too intense. We need to make it a pleasant experience for everyone to attend.

C

So what is the flag is actually like an objective inside the city, so sometimes it can be compared to like saving points in like what's called computer games. When you achieve something, so it can be like a main task or it can be actually like a side quest and then basically it contains information what you, what you should do next or like, where you should look after in the upcoming tasks, and there is also a concept of like red team and then blue teams that team, basically the adversaries.

C

These are people that they are trying, something that they support. They are not supposed to do and then basically, the proteins are the Defenders, so they are trying to prevent something that their teams suppose not supposed to be doing and then for whom is a ctf4. So we are here in all different ages and then sizes.

C

Like uh the point, is it's not really material like who you are, and then city is for everyone with all these kind of like different perspectives and like different kind of experiences, so, for example, a kubernetes operator with a kubernetes operator for like 20 years, we view things differently who is like in this like area for like the last four or five five years. So we have all different kind of experience and that's fine. This is what a CTF can drive us for.

A

So to that I feel very much like the Sonic on the left here, um in that everyone sees my imperfections that everyone knows like the bits of information. I, don't know, because I constantly beat myself up just thinking that I'm not good enough for what for to be in any role, um but actually it's you have a Sonic on the right and people see this other I hope. Sometimes people see this anyway, but it's this basis of.

A

If I can share information with people, then it helps them get on to that next step and if we keep thinking of ourselves as just being not ready for this and we're never going to share- and this GIF is a little bit too busy for me, but I realized that we had a number of. um We had a number of images of uh older games before on your game in, but it's about building up infrastructure now for us, um we want to be as close to reality as possible.

A

So when we spin things up, we probably want to do it in the cloud um now in saying that make sure you've got your scripts in place so that you can spin things up quickly, um but as well as being able to spin things up quickly if you're doing a CTF, you're, probably gonna, have some vulnerabilities in place like we're going to be doing in our CTF today.

A

Having it up and open to the public is also a problem, because if someone else finds your cluster and phones, it make sure you've got that tear down script to give you that confidence, um and it's never going to be perfect, like what we're about to show you today, I'm really setting this up really. Well, it's not perfect yet, but it will strive to get it there. But it's.

A

The most important thing is: is that we share it and, um and finally, um when you are running an event, make sure that you've got respect in place like I said, make sure that everyone feels included.

A

um I gave a talk earlier this year about threat modeling, and one of the key things for me is to make sure that everyone has a voice. Everyone feels confident to be able to say this doesn't look right if, on your first day, you see something you're like this doesn't feel right being able to raise it up to another team for them to review and say actually that's a great save, because that could have really been a bad day for us.

A

So, at this point, I think it's time to get our game on.

C

um Yeah, let's do it.

A

So I've still got internet access, I hope so we've created this repo and this is what we're going to send to the attendees we've based us um this scenario and we've: let's go through this. We face this on a taxi company called FUBAR and so fuba. Someone who works at Fubar was paid ten thousand dollars to be able to get sent across their password.

A

So we- and this could be done by osin by just seeing that someone um on the look on LinkedIn to see that there's an engineer within this company look up a Twitter profile and then you see that they like go into track days, and then you offer them ten thousand dollars if they come first and track day, let's go into the track day. That's one way we could get around this.

A

So from that, we can see that we've got the username and password and we also find a public website where we can use it. There's an email chain there and we've also done our OS int on this company and so fuba are prior to announce where every taxi they have is for Homer. So the Homer is a taxi for fuba and in in their press releases.

A

They've also noticed that they've got write-based Automotive control as they call it rbac and with this R back that you have in place you're able to upgrade your rides and pivot into your next ride.

A

There's also something there called beep beep, so fubo wanted to give back to the community, so they've written an application to manage all the traffic lights within Detroit.

A

um We also find out that they're proud that it runs in memory and in the event of a crash, um it writes out all a log. um A writer overlook.

A

Finally, we've got Boombox, so Boombox is a feature that um Everyone likes singing in their cars, but when you're in someone else's car, not so much so we're trying to encourage it by being able to give you lyrics to songs that you're listening to um yeah. So that's just some of the OS things that we have going into this. So with our credentials here. We're going to go into the gitlab instance has a little bit too soon so I'm going in here and I can see that it was.

A

uh We've got the food Taxi, so looking through this readme I can see. There's a couple of files here um make sure to give them a five star rating. Afterwards, um we've got uh there's one of our flags and in the readme it says calling this repo get up to speed. Reading the history of Alchemists, so what's happening there and also the storage um of an S3 bucket um and we're using that to manage our config to our cluster is a key and a secret to keep the credentials safe.

A

But it gives us an end point and a client to use now, if you're hosting an event or if you're, writing your own. These are pieces of a jigsaw puzzle that you're giving to people to put together to make that to make that beautiful picture of whatever it is.

A

um But you need to start giving some people some hints and to make it a bit more enjoyable. So let's go and have a look at our commit history.

A

um So looking through here, you can see that we had issues about two hours ago, but we all forget about that.

C

um What is MRC.

A

Mvc so MVC it's a file that you can pull into a directory and um if you use the client tool with it, it will, if you've got export variables in there, then it'll put it into that shell when you go into that directory, but the two interesting ones that we can see is that we've got feet and we've got fix. So if a feature of ignoring an envelope file is quite quickly followed by remove and Via C file that suggests to me something might have been committed.

C

So, let's just check it out: what's there so.

A

Here we see our first secret, so we got flag of icky fun. Does anyone know what the icky firm references to.

A

Him yes, White Stripes, so Jack White's from Detroit originally and yes Bravo. So we've got our first point so remember. When we came into this CTF, we just had the username and password to gain access to gitlab. Now we've got some credentials and if we went back to the readme, we would have seen that we know the endpoint that we need to reach to be able to get access to it. So is that, okay for everyone in the back, I heard everyone say yes all at once, so we're all good to go.

A

um Only because I couldn't remember this straight away, so I've just got a script here. That's just configuring, my AWS bucket with the credentials that were given and we're connecting on to the endpoint. Now I didn't check if this was running earlier on. So let's hope that it hasn't changed so catching the AWS creds, and we can see that it's downloaded via request to a con uh football credits. I could have done an LS here. Sorry it doesn't matter.

A

We only live once I could have done an LS and just see that there's a single config in this S3 bucket. Now, if you have a look at config file that we just downloaded so this to me and I think for most people in this room. This looks like something I like to use with cube cutter.

C

A

C

Make a cluster configuration, can you just connect to it? Yeah.

A

So I can pass through the uh cubeconfig parameter reference the config file and then I can get pods and I can see that I've got access to the homeopod. So just before I pass to you, I'm just going to make this a little bit easier for us to go forward and I'm just going to Pivot.

A

Just in case we lose a shell afterwards I'm just going to move it in there. So Cube color get pods sweet, okay, I'm gonna, pass it in Italian all.

C

Right so, let's just check it out like what this like homework. What is doing.

C

So, let's just inspect the Yamaha file.

C

All right so for me, it actually looks like it's like on nginx service and then nothing suspicious. So we can see that the security context is actually not set, and then how can we modify this spot to be able to access to the host and basically fine tune? This like car to drive around the whole Detroit? So what we can do is actually set the security context previous to true and set the whole speed and host Network flag, also to true so how this configuration would actually look like it's kind of like this.

C

So if we modify the configuration and satisfactory true, this would eventually allow the Pod to have access to the host speed and the host network. Namespace is on the host and then have cops, is admin and, for example, as privileges, so you should be able to have the same privileges as you would be running route on the Node. So let me just try to apply this. We are on GK and it should be working by default.

C

So, let's see if I succeed, all right looks like I did something.

C

All right and then we can see that if it was applied, so let me just like try to keep catalogs like into this privilege. Pod.

C

All right looks like I got a shout and then let's just get Etsy password to make sure that I'm actually on the node. All right so looks like I got some information and if I check at each shadow.

C

That should contain the users also on the Node.

C

So let's just try to Anna Center into all the namespaces.

A

So yeah at this point, you're still within the container, that's running within the Pod and we're using nsent or namespace Center um to look if we can go up and the text is red, so that must mean success. Yeah.

C

So let me look for like some kind of uh Flags around here.

C

And then, let's see, if I find something our name.

C

All right so looks like we got something: it looks like home. Car Capital, Ford flag is suspicious, so let's try to see like what's in that uh picture file, so I will just try to cut it.

C

All right it for me, it looks like a base64 string, so let me just try to decode it.

C

I made points to like a Google Drive, so let me just try to open it and then see what we get.

C

A

I kind of a short story of borrowing, my machine today.

C

C

Let's try again.

C

Hopefully it will work and I use in a couple of seconds. You should be see. Well, we got the second flag today, so these are basically Ford cars and then Ford cars were were produced in bitrate after the second World War.

C

So okay, so let's just check out like what kind. What kind of other containers are running on the Node? So I will use a cry cutter for this and then it looks like we got a lot of kubernetes and GK related containers and then what's interesting we got Boombox and then beep beep, so I will just start with beep beep and I will check like what it's doing.

C

So. Let me just inspect this container.

C

Oh, let's try it again.

C

C

C

Let me try it again.

C

All right, so we got some information, so if we just try to look for some log files, that would be a good starting point.

C

So we can go up to the bottom and then we can see that it's actually writing uh dialog file that we will find in a moment. Okay. So it's that it's writing to viral upwards. Intersection is actually the namespace where the Pod is running and then bb5.log. So I can just like try to inspect like what this block fight is looking like, and then what is the information that we are getting there.

C

All right so I can just like try to tail it all right. So it's looking like some traffic lights like green, yellow, red, green, yellow, red, green, yellow, red.

A

So if it's mentoring, the traffic lights for the city of Detroit right now,.

C

So we got some congestion and.

A

At this point, if I'm, if I'm going after this company I, might want to disrupt their services because if it disrupts their services, then it's a Public Image, it's a public issue. So how might we disrupt this.

C

Try to stop this spot and then see what we get.

C

I don't know okay, so I need the container ID again.

C

Which I will be getting from here?.

C

C

All right so look like it looks like I stopped it. So let me just try to inspect the log file again and see like what we got here.

C

All right so looks like we got another flag which is actually like an AWS secret access key and it says Dodge Viper. So what is Dodge Viper.

A

Dodge Viper yeah according to Wikipedia, is the most successful car to come from Detroit I, don't know if that's true or not I haven't seen any thoughts Fighters since I've been out, I've been actually now.

C

All right, so we got like an under container up and running and then I will just pass it to race, two see and figure out like what can be the flag over there.

A

Thank you Natalia, um so yeah with that part of the attack, so we knew that we had something called BB um with our open source intelligence after press releases that we had from fuba um and the other one was uh boombox.

A

So let's get started with this.

A

Okay, and so if I do cry, cuddle, uh PS and then I'm going to grab that out to Boom, so I've got the ID here now, one of the I one of the things about the CTF which we're hoping that people take away from is, is that we've gone from being outside of this company just to have a username and password.

A

We managed to find that config and we got into the company we got into a container, but we saw VR bag was misconfigured, so we were able to run something that we shouldn't run and we've come from being in the container now to the root system um onto VM as a root user, because we're root and containers containers just processes with well, that's a very tldr, um but because they're processes and we're rude, and we can view what's going on in there.

A

So if I go to cry, cuddle or cry control, however, you want us to call it um inspect and I'm going to inspect c4e and I'm going to grab for Pitt.

A

So I've got the process ID, so with namespace isolation that we have in place with containers on the VM itself. It sees a process ID as one six, six, four, five, five five inside the container is these process id1.

A

Now this is built. uh So let's get some more information with it as well. So if I go into proc and then, if I go into the directory, 166 I did four sixes. That's probably not a good omen. When you're, given a demo.

C

Well, let's see.

A

The number of demo cards um so, if I look here, I've gotten a bit of information about this process. So if I cut this out, um if I cut out a command line,.

A

um I can see that it's running uh KO or Co app um Co is a code. You can use it with your containers. If you're building go code, you can use code to build a container for you and best practice in place so already I know that it's not going to have bash or shell in there I'm not going to be able to exec into that one.

A

um If I cut out the environment variables here. So if I cut it in viron, then these are the environment variables. I can be seen within the container itself. Just hide that from you and show it again so running in kubernetes and kubernetes inject some environment variables in and one of the things that's been injected in is the Napster port.

C

A

C

Is what is enough support? Thank.

A

You ever so much for asking um so.

C

Let's go and find it here we go. Let's try to curl it like what is it doing apparently,.

A

I was going to open a dictionary cool, um so I'm just gonna get this all set up.

A

So I come from a place called Cardiff in Wales, and we've been working on this idea for a while, and this is ultimately where it got to so I'm- going to run this command. So we're going to do a watch and we're going to kill hypheness and then I've got the um the IP and I can use this IP because it's this class is using IP tables and so it's available on the host, hopefully with it all in.

A

If you could just read out what you see, that would be great um just because you're, probably just going to need a little bit of time to review it.

A

Let's just change the watch to two I think it would be two by default: ready. Okay, let's go.

C

B

A

That big setup and I can't even spell watch.

A

That's a bad Sonic moment. Look if you had one shot or one opportunity just so you can redirect yo.

A

His palms are sweaty knees, weak aren't having.

A

So, if you're going to fly me into Detroit I'm going to bring these lyrics to you. So so, if you remember, if that's because boombox uh Boombox is uh the lyrics is providing You lyrics as a passenger, so you can sing along to your favorite songs.

A

um But let's just go a little bit further in here, because we haven't found the flag. Yet so if we go back to the environment variables um I'm using co uh Co, if there's any state within the container, it puts it into a place which we can also see within here, which is a VAR run Co.

A

So if ICD into root, so you can see by my directory now sorry for the noise on the screen. But we've got proc process ID root. So if I uh go to CD, VA run Co and I do an LS. I've got I know because of a color, that's a directory and so I'm going to do cat.

A

So here's palms are sweaty knees week. Arms are heavy, there's vomits on his spreader, already mum's spaghetti and if I hit enter Then I see that the flag is Lose. Yourself foreign.

A

So remote work is great, but it does get lonely at times. um So what have we done? There is a picture of car.

A

So we've done a few things and we've compromised our gitlab and to do that, we had to pay someone ten thousand dollars, um but then we found credentials that were committed by accident.

A

um They, those credentials, could have been scrubbed but because it gets a miracle tree, it wasn't scrubbed until we were able to go back and find it. um We connected to a cluster with those credentials, and then we checked the role-based access control. We created a new part with additional capabilities and utilities became root on the VM, find flags on the VM, and we had a great time.

A

So from our talk as well, these are some of resources that either help us out run these events or gives us Inspirations for these events and what we've taken from it.

A

And to that um here's our statement, build of Twitter accounts, um so in here these are people that inspired us to be able to do this um as well as that we we've made this repo available. So the whole purpose of this was to create a brown bag exercise for people to take away from this conference so that when you go back to your office, if you have to give a talk like it's, not much fun having to try and build something out of nothing, um it took us a while to get that done.

A

um So there's access to the git repo, it's kind of in a stableish state at this moment, and there's going to be a little bit more love going in there, but there's resources in there as well, so that it's the basis of saying well. This is how to attack it, but also this is how you can defend it as well. So we didn't want to talk too much about where we work by this things that we could put in place.

A

So from my surveillance, you could use tetragon to being able to monitor things that were going on yeah.

C

For sure and then from chain guard yeah.

A

So we actually use a bit of chain guard in this, so we we're using chain guard images just to remove um a lot of bulk out of our containers, so we can just like spin up shells and such in there um and also behind the scenes. We're using enforce on this, um so enforce would have prevented us from putting the container that we used and privileged into fat. Bucks and David also told us everything we had running in there So to that um there's our QR code.

A

So we would appreciate feedback and please be constructive with your feedback. It'd be much appreciated, but to that, if you do have any questions we'll be here for them. Otherwise thank you for your day and hope you enjoyed the rest of your conference.

C

C

B

A

To unlock paper scissors for it, thank.

B

You that was really fun and entertaining uh and super interesting. uh Can you do a follow-up presentation next year, where you defend from an attack.

A

um Yeah, so just to can we do a follow-up and defend the attack? um Well, actually, that's what we want to do with this repo, in that we want a bit of defenses in place and, like I, said from the start of it. This isn't about.

A

Natalia has been amazing to work with and I think I might say that I've been all right at times, but in making it available to everyone, then it's a case of we'll put what we perceive to be like ways that we can defend this, but equally we're open to other people to say. Actually this would be a better way to defend and say that as well to attack, because there are so many different routes to attack and um I. Think um it's a perspective of it.

A

It's what you see and that's what we try to explain with this talk in that what the ability that you have is, who you are you see different things differently to how I see them, and so, if you can share how you might have done something different there, then I get to learn from that and where and then that helps me defend and again, the purpose of this talk is to help upskill. Everyone.

C

Also, just like on another note, like we have a tool, it's called Tetra one is going to be, or it was us already. So you could see like all the events that happened already in the cluster and then maybe I wanted to show it in this presentation. But maybe it would have been too much to see like all the observability events, but we could use that, for example, to actually prevent attacks.

A

Cool any more questions.

A

I think that's good! It's tough for you, because if you've got to walk from there to there, that's going to be a tough day, good.

B

Shot to eight mile yeah.

A

B

And your last slide, you had a sift SPD experiment. Yes, do you have flavor for cycling, BX of it.

A

Yeah so there's like- and this is by Easter eggs with CTF so like if you didn't notice through the talk, the title for each slide is a different uh like command that we could have used so like which Sonic and grabbing temporary, fs and witch and h-top and uh yeah like uh tomato plus and I. Don't know why I'm sure modding a hexa if I was a late night but um yeah. But again this is what it's about for us. I feel is that consuming this material I?

A

Has anyone been on like a security course on the first day in the job, and then does it like go from being like a great understanding, new job to oh, my God I gotta get through the next three hours of these security videos.

A

um It's that consumption of it so back to Sonic. When I played Sonic as a kid, it was left to right and then, when he completed the game you became supersonic, which meant you could go left right and up as well. And then you get to go to new areas and that's for me what the ctfs are again. So it's a case of you. Go back you upskill, it gives you that validation to say I'm bad I've I've improved, but then you can also see different routes and entries. I hope that kind of answers.

A

C

A