Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 4 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Beyond Kubernetes Security - Ellen Körbes, Tilt & Tabitha Sable, Datadog

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Beyond Kubernetes Security - Ellen Körbes, Tilt & Tabitha Sable, Datadog

Kubernetes is "the universal control plane," a "platform to build platforms," and its utility is more than just improved deployment and scaling. Your Kubernetes environment can support you as you level-up your security practices. In this blockbuster action thriller, Tabitha and Ellen will hack and defend Kubernetes for your education and amusement, including both commonplace and advanced attacks. They'll show ways Kubernetes can help you level-up your response to today’s challenges, including software supply chain issues. You'll leave inspired and ready to think beyond Kubernetes security.

A

Hey everybody, um you know like it like it says here, I'm I'm tablets able, and this is beyond kubernetes security. uh Thank you so much, especially the people that are coming in virtual, like I know it's really hard to do yet another virtual con, especially when you're like know that there's there's some people that are here in la it is wonderful here in la, but I'm so so glad, especially for those of y'all who couldn't make it for showing.

B

A

So let's let's get started, um does this look good? Can everybody sweep all right cool so um to move beyond kubernetes security? The first thing that we're going to need to do is understand, threat, morphology. So there's these threats that you already know about, like you know about internal threats, you know like.

C

Like a malicious.

A

Contractor or uh or a disgruntled, employee and external threats like the nsa, the gru, you know a board kid in ukraine: oh, oh gosh! What? What? What's going on.

A

B

Sorry, uh you don't know me yet, but by the end of this talk, you're presenting right now we're gonna be lifelong friends. I need you to know that the curse it wasn't really a curse, but I'm sorry this is going to make more sense. Later, I'm going to be broadcasting, you some things, I'm sorry, but they might be slightly harder. I'm.

A

So sorry, I I don't know what's going on.

A

This thank you. Thank you all for for putting up with this with this interruption.

B

B

D

A

uh Hey um yeah: this is tabitha, I'm I'm so confused ukraine. Oh, oh! No!.

A

I don't think I'm going to be able to do the presentation. I think we're.

B

Going to have to deal with this from the future telling me to use this thing to stand.

E

A

Yeah, I'm I'm I'm really confused too, but like I'm supposed to tell you about what happened when I was in pripyat in in 2013 and like I'm, not actually allowed to talk about that. So this must be really really serious.

B

Last try during person, but never thanks for the chance.

A

Not you me come back.

B

E

A

E

B

um Yeah, I'm gonna try to wipe myself.

A

Really you need to come back. This is this is so bad like we need to talk.

B

Hi, I suppose this is tabitha. I was told to talk to you, apparently whatever you were doing in 2013, um I was told it's happening.

B

I was told this would make sense to you and that he would help so whoever you are yeah. What's going on.

F

A

I'll help um hey uh so yeah. This is this is really awkward, but.

D

When I got out.

A

Of school back in the 90s, like was until just a few years ago, when I got started doing cloud native security, I was working on a project that I can't really talk about. It started out as a surveillance system called the heptagon network, but we we expanded it over time and it turned into like a multi-dimensional ai powered defense system called sky cloud.

A

With with the things that have been happening to you, I think, I'm afraid you're being targeted by sky cloud.

B

Okay, wait: I'm I'm getting all worked up because, like some pile of back scrubs, doing linear regression called sky clouds trying to actually.

A

There's: okay! That's enough! For.

B

A

Oh, no, no! No! You! You don't know what you're getting into you need to. You need to slow down.

B

Okay, look, uh I I don't know who you are, but um I've been doing this for a while. I know what I'm doing.

A

So have I and I'm telling.

A

No ellen, please I'm trying to help, but you're you're gonna need really serious planning or you're actually just going to get yourself in more trouble. Please call me back wherever you are.

B

Hey tommy, uh sorry, I'm doing garbage collection here on the whole time thing and the last worst house favorite. Can you forward her? Sorry, I mean me the me you. You were just talking to not me here that you know that me uh foreigner the third email you got from retuken in 1998.

B

and yeah just for the email, uh I'll route it to the right time and you know yeah the whole thing, but yeah just trust me that this is a lot easier for him.

F

A

Oh, my god, it's been so long since I had to read: japanese um okay so, like you were reading about buffer overflows and you found one in the login page.

E

A

And you wrote an exploit for the back end code, but the real login page is protected by input validation. So, like.

F

It's probably, okay, goodness.

A

uh That that could save us. I hope that after 25 years it still works. um I'm I'm gonna forward you, the email now um tell me, tell me if you get it.

B

If I just got your email, I didn't realize we're gonna help you with this. I said.

A

I'm gonna help you yeah yeah, I'm like, I think. If you run that code from that attachment inside the heptagon network, I think it'll be able to crash the the like sky cloud control system and, and I think, it'll save your life.

B

C

B

F

C

C

C

B

Thing that can only do like single commands at a time. I don't think we can run uh retsuko's search exploit with this. So can you at least help me with this.

A

I mean yeah. Of course I can help you with this, like not. Every vulnerability has an exploit. That's super convenient to use, but like put malware.

C

On the thing and give yourself.

A

Whatever you know.

C

Whatever quality of life things that.

A

You need, I don't think you even need anything fancy. Just like normal pen. Testing tools are probably going to be good enough.

B

C

B

We we got what you said. uh What now.

A

Well, you're in like congratulations, you win the game. Do the thing that you want to do? Look around the system figure out the situation escalate to root and run the exploit code. This thing's running on kubernetes, I'm sure you can figure out a way to be able to to be able to run something.

B

Oh, oh, it's a kubernetes system. I know this.

B

B

Operation not permitted.

B

Okay, fellow 69 blocked his class.

C

A

Of course it failed. I'm look I'm trying not to like. I know, you're under a lot of pressure right now, but like you could have checked out what was going on before you went and just tried to run. Some some exploits like their clusters, looks like it's really well configured and managed. I'm surprised they even needed a kernel update, but you know, I guess just patch is actually a lot harder than it sounds. Even for the heptagon and sky cloud.

B

Okay and what that is wrong,.

A

You should have checked out the situation more thoroughly like sure their colonel was vulnerable, but you got totally iced out anyway, like that kernel bug needs capnet admin to be able to trigger it, which you don't have.

A

Fortunately, a lot of mainstream linux, distros allow unprivileged users to spawn username spaces, and you can get capnet admin within your username space great. So now, all of that kernel attack surface is available to you, but there's an alpha feature in kubernetes 1.22 called set comp default. If you enable the feature, gate and the kubelet and then set the kubelet config that turns it on, then it takes advantage of some really good set comp filtering that jesse frizzell and some other fabulous hackers did years ago, like it blocks.

A

A few really weird sis calls that normal applications have no business, calling anyway like onshare, which is what you need to be able to spawn a username space, and so most applications can just run under it, but also it can sometimes shut down. Attacks like yours.

B

Oh um okay, so do I need to find like a different vulnerability that doesn't need that? That's this call.

A

Mean that's a thing you could do, but like it may not be that easy. Did you notice, you didn't, have any good linux capabilities in your bounding set when you looked at my container they've got their workloads pretty locked down like if they're, using really up-to-date things like they may have turned on pod security admission.

A

That's another alpha feature from 1.22 and it looks like they were ready to try it as soon as it was available so like if they're doing that, to enforce policy about what kinds of dangerous pop features can be used in what name spaces like you may struggle to find a pod that has enough permissions to do anything dangerous so like you, can keep plugging away at it and try and find another bug, but can you do it before they catch? You.

B

So what can I do.

A

I mean it really doesn't seem like you're going to be able to pound the front door down like this they're really well prepared for direct attacks, so I think you're going to need to attack them indirectly. Just try the usual things. You fish, a developer, commit some malicious code into their test. Suite wait for the cicd system to run the test code, easy.

B

Just commit some malicious code: hello, hello, yourself,.

G

Oh hi, what's up so.

B

Remember when we did this like that time,.

G

Ooh, do I get to get you out of trouble again like old times.

B

Well, yeah, like old times, so here's the deal yeah and then she told me to fish a developer. Basically.

G

I can't believe you're dragging me into this again.

G

Okay, do you know who works at the company? um I do not check linkedin.

B

G

Okay, ceo hr, that guy gandalf looks interesting who's. He.

B

G

B

This out, oh gee, hacker one of the last real hackers, and then he shares a link to a retrospective on the glory days of something something.

G

Sounds just like you used.

E

G

So I bet his whole networking setup is just like the good old days too. If you know what I mean.

B

Tabby, uh so we we got this guy, I I found him on irc. uh This is uh this is where his connection is coming from. I'm I mapped it, and this is what it gave me. Do you.

E

Know anything about.

B

A

Wait wait. What is that open, ssh 2.3? This is hilarious, that oh, my gosh, so there was this protocol vulnerability in in ssh1, where the the checksum that was being used was not strong enough. So, like you could cause collisions and decrypt the encrypted traffic like cool great, like that, was dangerous, and a lot of people caused a lot of trouble. So the developers put some code into the damon that would detect when that attack was being performed against the server and like shut it down. But you know how these things go.

A

The attack detection code also had vulnerabilities in it. Oh, my gosh, everybody was like hacking servers all day long with that thing back in the day,.

G

Found it, I think I found something: what do you make of this.

B

Attacks against open ssh security incidents, mailing aircrafts, okay, um open ssh, 2.2 upon something this is the crc exploit. I think this is what happy said this should do the trick now, if only we could get our hands on this puppy.

G

Wait hold up team teso 2001, that's didn't you get a whole stash of exploits from back then from that friend of yours. What's her face, oh.

B

Yeah you're right lex, murphy, dave's, sister you're right, give me a minute.

B

Sweet okay, uh cat.

G

Oh lord, it's in the write-up dummy right.

B

G

B

Is working? Oh yeah! Look at that! Yes, just a couple times! Okay, let me let me get into the code.

G

Doesn't work here um just be a good girl and do, as I say, script, slash, dev, slash null control, z, now stty size, stdy, raw.

G

You're welcome.

B

Okay, so did this: oh, it did cool um tubby we're in what do we do next.

A

What do you do next? You have access to their source code on the on the developers home network, like I don't know why they're letting them do that, but like good for us right, stick the kill switch into the test suite, let the build server run it and like live free another day. What just do it.

B

So this will make their tests run the two case binary, which they do and unsigned commit. Oh uh whatever, oh goodness, okay, still, stop stop! That's it! um Now. How do we check this? How do we? How do we know if this works.

G

I don't know check his email, okay, good idea um and of.

B

Course his favorite email client is.

B

E

So I guess now we.

B

Just sit and wait huh um in the meantime tabby. um Can you tell me how the zag spike actually works.

A

A little bit so like the main, fail safe for the sky cloud system was that all seven delegates of the heptagon network needed to be present to start it up once it's started, it can lose nodes and keep running, but if it's shut completely down, they have to restart it together and there's only one node left.

B

Okay, but who even are these people this? This is so strange.

A

Oh, you, you don't even know who they all are ellen there. There was the pentagon. There was like a couple of other countries, antarctica the moon and like nobody's really sure who the last two delegates were like. I'm, I'm not actually convinced that they were human.

B

Okay looks like we got mail cat, listen to this um you're commit something something has failed to build due to error and welcome. It's signature. Potential tempering detected counter measures active, of course can't this is. This is not.

G

Good, no, it's not uh what's your escape plan, there's because real talk you might be in actual trouble. Maybe I can escape plan.

B

B

I'll try something else I'll figure something out, but but can't you ow can. If, oh, if I get you in trouble again shut.

G

Up no I'll tell you when I'm out: let's keep going.

B

G

Let's keep going.

B

Tabby um things went wrong. What else can we do.

A

Yeah, I mean, of course, things went wrong. Nobody enforces mandatory, commit signing just because it's fun they're doing it in order to secure their software so like with their system set up to have signatures on commits validated in the build system.

A

Anything like what you're trying to do is going to be a dead end unless you can get access to the developer's private keys so, like I think, you're kind of stuck on this go after the other end go after delivery like if you can push a malicious container into their repository and then overwrite existing tags for containers that they're using then like the next time, a pod dies, or they scale up or something like that, like they won't even know the difference. They'll just slurp down your new container think it's the.

F

A

Run it and then you win.

B

Okay, so I was looking around and I I think we got a good candidate. Let's try hijacking the images for this database maintenance operator. What.

G

The hell build system.

B

G

B

It's a build system, it looks more like a development. Environment is code kind of a thing, so it looks like here's where it makes the call to create the images and here's the actual image it generates.

B

Okay, let's start this thing up: oh wow, this is cool.

G

So easy, even your attacker can figure it out. huh Yeah.

B

Well, at least I can um okay, let me add um that code in here.

A

Okay, yeah yeah: let's do that build that thing.

B

Okay looks like we're done here before I ship this tabby. What did I do rob this time.

A

I mean for once. I think you got it right, like you tested your changes locally, so you know the container will work when they run it push it in there. Like congratulations, you're learning,.

B

Cat, um so we gotta do the devopsy stuff. Now, um how do I do the you know the image switcheroo thing.

G

So what you gotta do is overwrite the image tag right. First, get the id of the image that tool just built for you with docker images, then tag it with the same name, but with the remote registry.

B

G

Do it with docker tag the image id, then the remote registry slash the image name.

B

G

Yep now check it with docker images. Good and off. We go docker push remote registry, slash tag, oh my god, and there you go. You did this.

B

Okay, how do I check um how do I save the thing.

G

Check their observability dash, look for any logs from db maintenance and see the brand okay.

B

So beta dog logs uh jp maintenance yeah. This is what we wanted. um Well yeah keep control of energy logs strategically sync sync failed with admission webhook container failed signature, validation, potential, tempering, oh counter measures activated. Oh oh.

B

Can't I think this is when you tell me around.

G

Yeah, I'm sorry it didn't work.

B

Yeah um cat all logs delete our pictures, delete like hell. I don't know mixtapes um everything it's safe for you. If.

B

If we never mind.

G

B

G

I hope you get out of this okay.

B

B

Tubby, what can we do next time, I'm out of ideas.

B

A

Oh, oh, my gosh, I'm I'm I'm so sorry they're their defense is really cutting.

C

A

There's so much work going on right now around ensuring kubernetes clusters only run trusted. Images like I was just at kubecon and, like I swear half the talks are about software supply chain security like people really are thinking about this, like, for example, there's an admission controller called connoisseur that can check container image signatures before it runs before it lets them run. um Projects like sig store, tough and in total, we're making it easier than ever to like ship things that are signed so that they are validated some.

C

A

Are building on this work with like even more sophisticated private tools that maybe we'll seek it open source someday? Of course, the heptagon has integrated custom container image verification into their kubernetes clusters like you're just caught. I don't think there's anything else we can do unless you could like send yourself back in time before they were so well defended. Like I'm sorry,.

F

I'm I don't know what I can do to help you.

B

Hi debbie, uh I'm sorry, it's it's been a few weeks since you last saw it anyway.

H

A

uh I was just joking about that time, travel thing but like cool, let's do it.

H

H

B

Transformation.

A

uh If you can modify their legacy code, that's still running, even though it was written back in like 1997, I maybe you would be able to get those changes in well before any of this focused on security started happening.

A

Maybe today their defense wouldn't be able to save them.

F

A

If that's a thing you can do, though,.

B

The math took a little bit to figure out, we did it and by we I mean lucas did all the work so.

B

So it's gonna be tricky, but there's a chance.

A

Good good luck, ellen.

B

E

B

Calls uh I need to go call myself in the beginning of this whole story. Even so see you in 1997, and hopefully I can uh so I can make.

H

H

On this event,.

E

A

Oh, my god, they're really in 1997.

B

Here I am it's 1997 and uh yeah: I'm ready.

A

I I I can't believe it um wait, wait I'm worried they might somehow find this bug in the like 25 years between then and now. What if you made the kill? Switch mechanism only appear for a couple of minutes like during kubecon.

B

Oh, I can only appear for two minutes: yeah. That's that's a good idea: okay, yeah yeah. Let's do that.

A

Oh, we shouldn't do that, though, because like the schedule can be weird and like what, if my talk runs over and we don't have the timing right, um I don't know you probably shouldn't.

B

Tell me I am literally streaming to your exact time right now. I know what time it is.

A

Okay, if you're, if you're sure.

B

I'm gonna hang up for good. Now, I'm gonna do the hack, you're gonna have those two minutes to get in there. After that, I'm gonna have to label for a while you're gonna know if I made it out, okay or not by, I don't know check my twitter see you back in the future. Maybe not goodbye, guess I'm gonna start by connecting to their network and downloading their repo.

B

So here they are. What can we do here? Let's see source code, libraries login, yeah, okay. This is what I want. Let me use these libraries, so I'm going to fill the buffer that ritsuka had talked about here.

B

Filling hashtag with a large pile of garbage and okay. If this crashes, it's gonna crash sky cloud, let's see.

B

Yes and I'm back to the future.

E

A

I I think they're alive.

A

A

All right, like real talk, I don't actually know what time it is, but I know we were right up to the lunch break so like can we do questions? Do we have questions.

A

Yeah, I guess we won't have official questions but like I'll, be happy to put my mask on and and talk to folks. Thank you all again so much for coming. Thank.

B

B