Kubernetes Kubernetes Contributor Summit - NA (San Diego) 2019, 26 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Contributor Summit NA 2019: State of Kubernetes Security

Description

Last year we presented on the state of Kubernetes Security across the past, present, and future (https://youtu.be/WcJCBOIMZkE). This year, we'll continue that tradition by:
- Reviewing the major security vulnerabilities and milestones of 2019
- Reviewing progress on the Big Problems we highlighted in 2018, and highlighting other work in progress
- Looking forward to 2020 with predictions of major themes and open issues

Event link: https://events19.linuxfoundation.org/events/kubernetes-contributor-summit-north-america-2019/
Session link: https://sched.co/VvMj

A

All right. hello, everybody.. Thank you for coming to listen to tim and me. Talk.

A

I'm cj, this is tim.

A

tim's, a co-chair of sig auth in kubernete's and then we're both on the product.

A

Security committee.

B

Nope, there we go. okay. yeah. so, uh,. If any of you are at the, at the summit last year,.

A

You might be wondering, hey,, didn't. I already see this talk? and yes, you did.

A

Um, we gave a similar talk last year,, but uh,. It's 20,.

A

19 and there's more new stuff, and it turns out that not everybody has all the.

A

Free time to attend every single sig, auth meeting and parse through every single.

A

Line of all the release notes that we've had over the years. uh,, so we're gonna,.

A

We're going to give you the spiel for this year.

A

Of what are we going to tell you about? we're going to start off kind of reviewing the cool things that happened in.

A

2019, um,, including stuff that maybe you might consider not cool.

A

Cves, although they're fun to talk about too.

A

We're going to go through some of the things to be aware of some of the hot topics, the,, the interesting stuff that is currently going on in the security space and.

A

Finish up with a kind of,, what can you all do to help us out.

A

So a lot of you might have heard about this product security, committee.

A

We used to be called the product security team., we're now a committee which gives us an official state in the kubernetes.

A

Landscape. um,, so pse is responsible for incoming disclosures about potential vulnerabilities.

A

And then managing the fixing release process and comms, uh, as we,.

A

As we disclose those and send out patches. uh,, so we've had some new members on the team, uh, this past year. Just for zelle,.

A

Stepped down from the team. and, uh, brandon, brandon phillips, is also making plans to relieve himself of some of his.

A

Responsibilities., a lot of the structure and process we have today is due to brandon's, diligent.

A

Hard work. so we'll definitely miss him,, but he's, he's earned his, uh,.

A

A

So this year we added a official on-call process for our product security.

A

Committee., that means there is a person that is actually tasked with responding to those.

A

Incoming emails. and we do get a lot of emails and they're, mostly not spam.,.

A

Uh,, so people are looking through kubernete's trying to find out things that are.

A

Wrong and could potentially lead to security vulnerabilities.

A

And so out of the,, what do we have? 159 reports, uh,.

A

We actually had 15 cves that we,.

A

That were graded out as medium or higher that we sent out disclosures about.

A

So here's, here's the list I went through and I tried to grab all the medium or high cves.

A

That we uh, that we dealt with this past year. uh,.

A

I'm not going to talk through them all,, but I encourage you if you have time or interest, go ahead and look through these on your own time,, both as a fun little project and to help us think through the places where these.

A

Bugs tend to hide., and maybe you can come up with one that'll end up on this list for 2020.

A

I am going to talk about a couple of the interesting ones we saw.

A

If you're hanging out on the security announced list,, you probably saw several iterations of this kube control, cp bug.

A

So it was a client side, file, modification vulnerability where, if you ran kube control, cp against a.

A

Potentially malicious container,, the tar ball you got back, could actually expand a symlink outside of where you.

A

Are extracting it and caused some files that you were not expecting to get.

A

Modified to be modified., so we fixed the bug and then it turned out. People were fairly ingenious with.

A

Coming up with new ways to trigger basically the same behavior,.

A

Combining sim links and uh,.

A

Playing games with the order of operations within cube, control. cp.

A

Fairly happy with what we ended up with for a solution. uh,, we think it's pretty solid,, but in one 16, we basically just said no sim links in cube control, cp.

A

So easier fix there. the other sort of class that we saw,.

A

Uh, come up this year, a good amount of times where these,.

A

These das vulnerabilities., so this first one, uh,, the jason patch library, cube,.

A

The cube api server supports the jason patch content, type,.

A

Which allows you to send a fairly concise payload that indicates to the api.

A

Server,, here's the changes I want to make to this object.

A

they might be fairly complex, lets you do a lot of interesting, stuff., uh,.

A

It turns out that interesting stuff can also be,.

A

Let's make the api server run around in circles for a really long, time. uh,.

A

The second one was a bug or a,. You know,.

A

Potential inefficiency in the.

A

Basically, throw a bunch of pings and resets at it and you can make it allocate.

A

Enough memory to eventually fall over.

A

last one was a [inaudible] similar to the first one,.

A

A bug guillermo has this awesome: aliasing, feature, uh,.

A

Where you can indicate that you want to repeat the same object, a bunch of times.

A

Um, turns out, if you nest, those enough,, you can get up to a billion fairly quickly. uh,.

A

So all three of these had a couple of things in common.

A

they were dos vulnerabilities., so we're starting to move on to the world where we have to consider that the.

A

Cube api server has expectations about being up for, uh,.

A

For its various tenants. they're also, uh,.

A

Treading into the world of dependencies., so each one of these were we had to manage process of actually updating a piece.

A

Of our core of our dependencies and kubernetes and getting that done and then.

A

Getting that rolled out., so luckily, with each of these three,, we had a thoughtful, responsive maintainers that worked with us to get these in, in a, in a,.

A

In a reasonable amount of time. uh,, that may not always be the case.

A

And so tim will have more to say about our dependency situation. Later.

A

Anytime, we start talking about dos,.

A

The question comes up about this anonymous off flag, in,.

A

In kubernete's., so anonymous auth is a feature that allows anybody to talk to your kubernetes.

A

Api server. and if they don't present credentials,, they flow through the system. As part of this magic group called system.

A

Unauthenticated. uh,: this is useful if you want to have a healthy end point where a thing that is.

A

Doing your liveliness probe doesn't have to have credentials to be able to talk to it.. It's also potentially dangerous, as we see with some of these vulnerabilities.

A

That that are made worse by being exposed to whoever can talk to your kubernetes.

A

Api server., so my opinion is that anonymous author is a valuable in a feature in the good.

A

News is that we harden the default bindings in one 14, so we reduced the default.

A

Set of api's that are open to this anonymous access., a note though,.

A

This applies to newly created clusters in one 14, so if you created your cluster.

A

Before then and you've been upgrading, it along, maybe take a look at applying these newer bindings.

A

Another cool thing that happened this year,, we had our first third party security, audit of kubernete's.

A

The big things that came out of that. we've got a threat model and we got a pretty thorough security assessment which.

A

Came up with some potential vulnerabilities for us to go fix in the code base.

A

If you want to learn more about that,, there's a talk tomorrow and another one wednesday at the, in the main track.

A

Another thing which I'm really excited about these service accounts.

A

So if you've used kubernetes, you've used service, accounts,.

A

You know your pod ends up with this jwt token mounted as a secret,.

A

You automatically use that token as part of the client libraries to get access.

A

To the cube api server.

A

there's a couple of downsides to this legacy way of dealing with service account.

A

Tokens. the tokens don't expire and they're, not scoped., so if,.

A

If one of these escapes,, you kind of have to notice they're,, they're rotatable,,.

A

But you have to notice, and you have to do some work..

A

They are also usable against your kubernetes api server from anywhere,.

A

Which means you don't necessarily want to use them for external integrations.

A

So we don't have this new way built on top of the token request, api.

A

Uh,, instead of pre-filling out a secret and then mounting that c,.

A

The tokens are generated on demand by the cubit who mounts them into the pods.

A

That request them. those tokens have an expiry and they have an explicit audience,, which means that tokens for your cube, api server, are,.

A

Should be only usable there. and you can request tokens for other things which allows you to do cool things.

A

With integrations. now,, this all happened last year,.

A

All the code went in and I think one 12 to make these features exist.

A

So why am I talking about it? Now? uh,.

A

We actually got some usage out of this in 2019, so in one 14 we migrated some of.

A

The core controllers to start using this flow.- that was a really big step.

A

And then there's at least two cloud providers I know of that are using these new.

A

Tokens as the basis for their identity integrations with their cloud api apis,.

A

Which is really cool to see this uptake. and with that,.

A

I'm going to hand it off to tim to talk about the node restriction, feature.

B

Thanks. uh, yeah,, so in one 13 we started scoping down a bit more. What nodes are allowed to do.

B

Specifically cubelets. um,, we talked about that last year,.

B

But we've continued to kind of iterate on that. um,.

B

And continue to roll out more label restrictions. and one 17,.

B

We added restrictions on owner, references., um, and there's a proposed cap.

B

It was originally targeted at one 17,, but probably hit one 18 to restrict what mirror pods can be labeled with.

B

Um,, but as all this enough, um,, I'm just going to shamelessly plug my talk tomorrow, where we'll do a deep dive.

B

On node isolation and some of the other problems outside the node restriction.

B

Controllers. um,.

B

So let's talk about some hot topics., uh, kind of switching over to a current things that are going on., um,.

B

So pod security policy fills an important hole in it. By restricting what can.

B

Actually run in your cluster. uh,, but there are some problems with pod security policy, today. uh,.

B

The first is a sort of a flawed authorization, model. um,.

B

The way it works is you can bind a policy to a user who can use that policy or.

B

You combine that policy to the pod that you're creating service account user,.

B

Um,, which is kinda, confusing. um,, and it turns out the latter way- is actually the right way of doing. This. um,.

B

And if you go the first way, you can get some unexpected breakage later on.

B

Uh,, it's really difficult to roll out pod security policy because it's a global.

B

Configuration for the whole cluster. you have to have policies enabled on every pod. Before you can turn it. On.

B

Um, and it's also kind of grown over time., and so it's got this kind of inconsistent api, uh,, which is, you know,.

B

We can clean that up,, but it's also sort of unbounded.

B

We don't have a good definition of what belongs in pod security policy and what.

B

Should be delegated to other policies.

B

And so with all of that, pod security policy might never make it to ga.

B

This is something we've been talking about in sig, auth, lately., um,,.

B

And trying to figure out what is the future for this important, uh, policy api.

B

Some alternatives, we're exploring are gatekeeper or caverno.

B

These are different plugins through the admission webhook api,.

B

Um, that kind of generalize this idea of policy. um,.

B

And also template to make it a little simpler. um,.

B

David eads proposed, uh,, integrating openshift security context constraints into upstream kubernetes.

B

As a alternative to psp is going back to psps roots. um,.

B

And we've also talked about some builtin minimalist profiles.

B

So this is all under active, discussion. um, and we'll be going deeper into this in the sig auths session on thursday.

B

So if you want to talk more about that,, come and join us there. uh,.

B

The image promoter, um, just to show of hands, who here has heard of the image promoter? okay, a few of you.

B

So, uh, right now, the way kubernete's production, container images get pushed is,.

B

Uh,, if you do not work at google,, you find a googler and you ask them to push the image for you. um,.

B

So, obviously, for a open source project like kubernetes,.

B

This is less than ideal and the image promoters trying to solve this problem.

B

So the new way is, you have a sub project owned, staging repos,.

B

And so a contributor to that sub project can get access to push their image.

B

Directly to the staging repo., then they submit a pr to the image promoter manifest,, which says,.

B

Copy my staging image over to the production repo that gets approved by a.

B

Project maintainer. and then the image promoter bot pushes it to production.

B

So why is this a security?? Why are we talking about this in a security talk? um,.

B

The, uh, we've been gotten away with being a little lazy with how we build and push.

B

Images. um, in part, that's just been leaning on google security to assume that it's okay.

B

Um,, but as we open this up to the world where anyone can potentially be pushing.

B

These images to production, um, that kind of worries us a little from a security perspective. um,.

B

Kubernetes doesn't have a security operation, center. um,.

B

And so as we open this up, we're also going to be working on migrating towards a more, um,.

B

Uh, kind of cic d, uh, oriented system where, rather than building these images on individual developer,.

B

Computers, where, uh, building them through a cicb pipeline.

B

And so a person doesn't actually need to do the push at all.

B

And really we should've done this a while ago. so happy to see it happening. Now.

B

Um, an update on the bug, bounty. um,- I think we mentioned this last year that it was coming or that we were talking.

B

About it. um,, we now have the vendor, we're working closely with hacker one.

B

We have weekly sync ups and we've launched the program in a sort of private.

B

Alpha state. we've got 12 submissions and we've awarded one bounty.

B

That probably doesn't sound like a lot,, but it's actually been really helpful in figuring out,.

B

Helping us fine tune the policy for the bounty and figure out how the process.

B

Works., so we're excited to launch that sometime in q, one in 2020.

B

All right. now for my favorite part of the talk,, when I get to complain about things.

B

So the first is vendor dependencies.. We mentioned this last year,.

B

We called it out as a problem that we have a lot of vendored code. That's kind of.

B

Stale. and unfortunately, the situation hasn't gotten a whole lot better, since then.

B

So last year was around one 13.

B

We can see that actually our vendor dependencies have continued to grow since.

B

Then., if we consider.

B

Golang.Org/X an extension of the standard library,, then it looks a little better but still sort of flattened out.

B

This is a concern from a security perspective, because one that's a lot of attack.

B

Surface. we have twice as much vendored code as actual kubernetes code and although a lot.

B

Of that vendored code is coming from projects with that are well maintained and.

B

Have good security posture,? We don't know what the status of all of those vendor projects are.

B

And even if it is a well-maintained project,, we don't always do the best job of pulling that code into kubernete's and.

B

Keeping track of security issues in those projects. by the way,.

B

Just to mention that spike, you see in kubernete's around one dot, five that was.

B

When we started checking in a lot of generated code and then realized, that was a.

B

Bad idea and pulled it all out.

B

So yeah, sig architecture has a code organization,.

B

Sub project- that's thinking about this., um,,.

B

There's also efforts to pull out some of the cloud provider dependencies and.

B

Eventually, I'll, all the volume vendors dependencies as we migrate to csi.

B

So I'm optimistic that things will, that that graph will start trending down in the next year.

B

So secure defaults., we talked about this one last year too, and we made a little progress,.

B

But it's still very hard to change defaults and a lot of the problems that.

B

Existed a year ago,, we still have that kind of problematic state set comp as an example that gets.

B

Brought up all the time,, but there's many others as well.

B

Um,, and so if we haven't made a whole lot of progress here,, maybe we should talk a little about who's responsibility. It should be to.

B

Actually fix these defaults, if we do it in core kubernetes., this is kind of,. When you think of secure default,.

B

Think of,! Well, we should change it in the upstream project.

B

That is obviously nice, because we get the maximum reach and consistency by.

B

Having the defaults in core kubernetes., but it's also the hardest to roll out.

B

We don't always have a good idea of who is exec,, who is running what in the open source project.?

B

We don't have that dillon wrecked line to the, to the users and the customers in the same way that, like a vendor, might.

B

Mmm., so maybe it falls on the distributors,. The cloud providers, cube, adm,.

B

Different cluster bring up tools., there's more power and maybe even more incentive to roll out secure defaults.

B

Among the distributors., but it also fragments the ecosystem,.

B

Especially if those distributors don't agree on what the most secure default is.

B

Or what the right default is.

B

I actually think that this is inevitable..

B

I think if we don't fix this in core kubernetes,.

B

Then there's going to be more and more pressure on the distributors to set the.

B

Defaults, right., um, and kind of fix the perception there.

B

And so my concern is that if that's inevitable, then are we going to inevitably fragment the defaults in the default state.

B

Through that?, or maybe it falls on the users.

B

This is sort of the current state and arguably not actually what it means to be.

B

Secure by default,, but it sort of falls on an ecosystem of consultants,.

B

Tools and vendors to help users get that default. State. right.

B

Um, so yeah, so kubernete's, security, um,.

B

B

we have this product security committee that we're at, like, I think, seven full.

B

Members. um, and we've launched a bug: bounty. we've are,.

B

We're launching the bug bounty. we've got this audit thing that we've got a lot of great results. From.

B

Um, and so yeah,, I would say that the project is maturing nicely from a security, perspective. um,.

B

I think misconfigurations are still the primary pain point from a security.

B

Perspective and the main source of compromise that we've seen.

B

So I'm hoping that this will be a big theme going into 2020 is trying to get to.

B

A better state of secure, defaults and secure configurations and try and fix some.

B

Of this perception, that kubernetes is not that secure.

B

Um, and then in scare font we have supply chain.

B

This is something that it scares me because I don't feel like. I have a very good.

B

Understanding of the problem. um, there's a lot of different work going on in kubernete's across a really big.

B

Surface area- and I don't think that we have really good visibility into every.

B

Piece of the supply chain., so I think this is going to be something that continues,.

B

Be a theme for security going forward.

B

So, if you'd like to help out with any of these efforts,.

B

I've put up a bunch of information here on how to get involved.

B

The security process. uh,- this is sort of what we do in the product security committee and responding to vulnerabilities. we're not looking for new members for the security committee right now.

B

We need to keep that small because of the sensitivity of the information we deal.

B

With. but there's a lot of good ways to get involved. There,.

B

Specifically, labeled a bunch of issues as help wanted, um,.

B

If you'd like to help with the security process, um, fixing, that. uh,,.

B

The security audit working group is getting ready to do another round of audits.

B

Of the, of all of kubernetes. um, and they're, looking for volunteers to help out. uh,.

B

If you're interested in the supply chain problem,, the community's infrastructure working group owns the image promoter and they're.

B

Doing a lot of great work around that. and then of course, release engineering and how we build releases for dependency, management,, uh,.

B

That's being owned and driven by the code, organization.

B

sub project of sig architecture and sig, secure defaults, sort of spans the whole.

B

Organization., so that's what we have. thank you.

B

And there's a link to the slides at the bottom there. if you want to go back and pull from any of these links.

A

Got one over here. mo at least entertain us.

C

Uh, so on the vendoring thing,, so you know,, I've been following, like jordan, liga did the work to move us to go mod or start.

C

The process with piles of bass, scripts. um,.

C

At the like bare minimum,, should we try to like forcibly like update all of our depths to like the latest.

C

And greatest and see if, like cis still passes.,.

B

B

is there anyone from code organization who wants to talk to that point?.

B

Um, I can take a stab at it. um,, so we're starting to do that with some of the dependencies. um,.

B

I think that actually the.

B

Golang.Org/Ex dependencies that are kind of, sort of pulled out of that graph there we are updating with every community's.

B

Release. um,, it's hard., one of the,.

B

Some of the problems we have with dependencies today are there's no clear owner.

B

Of the dependencies, like who knows what this thing is actually supposed to be: doing. um,.

B

And so, if we automate and we don't, so combine that problem with the lack of test coverage,.

B

I don't have a lot of confidence that I can just update this thing to the latest.

B

Version and not only not break anything but have confidence that I didn't break.

B

Anything. um,, so I would definitely like to see us work towards that. but, um,.

B

I think that just blanket updating everything to head, uh,.

B

Would probably have some consequences, you know. so.

C

When I was working on open shift,, one of the, we,.

C

We kind of gave up on this problem and we basically said that,.

C

Um, [inaudible], if you know what you want for some dependency,, like a branch or a version,,.

C

Find you stick that in, glide. um, everything else,.

C

We're going to force update the master,, the food breaks, go fix, it., right.

C

And it made bumping dependencies,.

C

Extremely painful cause you would get random breakages and you know,.

C

Really large diffs., but the flip side of it is. We were really current on things., so.

C

Mmm. there was like some random, like h, p, two bugs that we never hit because our,.

C

We'd actually, like accidentally inadvertently, like bumped, that library far.

C

Enough that we had gotten the fix and before, like we realized, we actually needed the fix. so, you know,.

C

There was some benefits here and there,, but maybe that's too aggressive for, like a very large, open, source,.

C

Mmm. project., that's not a product, that's meant by one company, like red hat did.,.

C

Um, I,, I just wonder like, to a certain degree,.

C

Like certainly like, we get decent, compile level errors when like, you know,.

C

Just the api of a repo changes. um, obviously,, you know,.

C

You can change the body of a function to do something completely different and leave the inputs and outputs. Looking at exactly the same. um,.

C

And then sad things, happen. um, yeah,, it's good.

B

A good point that we should follow up on.

D

So the, uh, discussion around getting rid of part security policy, and, uh,.

D

Introducing something else. uh, so are we gonna just include something as a controller?.

D

There are,, I mean, what I'm,, what I'm trying to get to is,.

D

Are we going to have something from the outside or do we want to, uh,.

D

Build it inside the kubernete's, because the, all the things that are listed in the slide that, I mean we can still use them, today. right?. So what's the, uh,.

D

The committee thinking over that? um,.

B

So this is still definitely an active area of discussion. um,.

B

The current direction, we're leaning, is towards pulling pots or pulling pod.

B

Security policy out of core kubernete's,, um, and using,.

B

Uh, web admission web hooks basically. um,, so there wouldn't be a core solution there.

B

we won't do that until we are confident that we have a good recommendation and a.

B

Good migration path to something outside of core.

B

Um, [inaudible] but uh, yeah, come to [inaudible] and talk more about that and uh,.

D

Abode the security force. uh, so like as a, as someone who is working in the upstream, uh,.

D

Like gantry, like,, isn't it this,? Isn't this the right time to, you know,.

D

Force the secular defaults?, even if it's a breaking change,.

D

How much of the registers do think, uh, you'll get an at what levels?.

B

Um, yeah,, it's a complicated question. um,.

B

I am certainly biased. Coming from the security perspective, towards, uh, yeah,.

B

Let's,, let's do it and just, uh, it's going to be painful at any point.

B

Um,, but we definitely don't want to just break users. um,.

B

If we make it too painful to upgrade to the security faults, uh,.

B

Then what happens is people just don't upgrade. they say,.

B

I know you're going to break me,, so I'm just gonna stay on one 12 for four years.

B

Uh,, I'm just gonna stay on this old version and not get all of the security patches.

B

That are going out. so that actually ends up leaving users in a worse state. um,.

B

So it's a,, it's a pretty complicated question. um, and how can we,.

B

How can we do this in the least painful way,, but still end up in a more secure state?.

A

It's especially tough, considering the broad variety of users,.

A

Types of users we have., we have some people who would be fine if we just broke them. Every time,.

A

And then they could quickly catch up.

A

We have other users who reacting to that sort of thing is a,. You know,.

A

Company's stopping process., so it makes it difficult to do just at the one level.