Flatcar Container Linux Tech talks, 7 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kinvolk Tech Talk: Securing Kubernetes addon components (kube-rbac-proxy) with Frederic Branczyk

Description

Frederic Branczyk, software engineer at Red Hat, stopped by the Cloud Native Computing Berlin Meetup on October 25th to discuss kube-rbac-proxy.

Kubernetes RBAC is great for authorizing requests to the Kubernetes API, and is a vital component for a secure Kubernetes cluster. Wouldn't it be great to use RBAC to authorize requests to your applications as well? In this talk, Frederic showcases kube-rbac-proxy, the tool he developed just for this use case. While operating his own Kubernetes cluster, the need came about to protect applications that don't have native authentication and authorization means to utilize the methods available by Kubernetes.

Speaker: Frederic Branczyk
Frederic is an engineer at Red Hat (previously CoreOS) contributing to Prometheus and Kubernetes to build state of the art modern infrastructure and monitoring tools. He discovered his interest in monitoring tools and distributed systems in his previous jobs, where he used machine learning to detect anomalies indicating intrusion attempts. He also worked on projects involving secrets management for distributed applications to build sane and stable infrastructure.
Twitter: https://twitter.com/fredbrancz
Website: https://brancz.com/

Meetup: Cloud Native Computing Berlin Meetup
Event: https://www.meetup.com/Cloud-Native-Computing-Berlin/events/254884066/

Host: Kinvolk
Web: https://kinvolk.io
Twitter: https://twitter.com/kinvolkio

A

Okay, I'm frederick. Yes, I used to work at korres.

A

Acquired earlier this year, if you want to ask yes, it's great reddit reddit is a.

A

Great company to work for all my team is still there, so there's that yeah. So what.

A

Do we want to talk about today, so the coop arabic proxy is something that I.

A

Started putting together at coop con in austin last year.

A

And I was talking to a couple of now colleagues about this and they said yeah.

A

This is really cool and then I ended up giving. Basically, this talk as a.

A

Lightning talk at coop con copenhagen, where I really pushed together, 15.

A

Minutes of content into a five minutes, so today we'll hear the non full speed.

A

Version of that, so really what? What did we wanna us or what did I want to solve.

A

With this problem, so I work on prometheus and also on kubernetes things.

A

And a really common thing that we have is that we want to protect metrics.

A

Endpoints that prometheus scrapes from exporters or various metrics sources.

A

Right so that only prometheus can actually see this data, and it turns out.

A

That kubernetes already has some mechanisms for this that prometheus.

A

Already has to be configured for in order for prometheus to, for example,.

A

Collect metrics from the api server or from the couplet, so this is really a.

A

Recurring theme and if we're already running within kubernetes, why not reuse.

A

This mechanism that already exists, but before I jump into some of the details, I.

A

Just want to give a very quick primer on on our back, so our back stands for.

A

Rollback role based access control and really what that means is that we.

A

Declaratively, like everything, incra nettie's, define a role. So a generic.

A

Thing that when someone has this role, this is the things that that entity can.

A

Do so in this example, I have some some some name in there and I see.

A

By an api group- and I say within that api group, which resources can I access.

A

And if I leave the api group empty because of legacy grantees things, that's.

A

The core api, so pods replication controllers services and all those.

A

Things- and I can only get and list them and I actually didn't properly update.

A

That name because it was supposed to be the view role and then there's a role.

A

Binding that actually binds the role to to an entity. So in this case I bind it.

A

To a service account and a service account is really as the name really.

A

Says is an account that is supposed to be used by a machine entity, and there.

A

Are there are so users and groups, but in this case I just used a service account.

A

So now that we kind of understand how our backwards and what our problem set.

A

Is, let's have a quick look at how kubernetes already does this without the.

A

Prometheus part, so there are two two things involved with with our beck or.

A

Really arbic is only the authorization part, but in order for us to be able to.

A

Perform authorization we need authentication, because otherwise I could.

A

Just say I am a cluster admin and a cluster admin obviously has rights to do.

A

Whatever, but we want to make sure that prometheus only has the access that.

A

Prometheus is supposed to have so there are really a couple of things and the.

A

Quark see today implement implements three authentication mechanisms. These.

A

Are basically taken exactly from the communities api server because it has.

A

Nice modules and they're reasonable and those are client certificate.

A

Authentication so just normal mpls, as michael already demonstrated for the.

A

When components within communities are supposed to authenticate against each.

A

Other that's a good choice to use, but there are also service.

A

Account and the nice thing about service accounts or, more specifically, the.

A

Service account token that is issued for a service account.

A

Is that there are already rotation mechanisms, which is, if you've managed.

A

Infrastructure, that's kind of a kind of a nice thing to have when there's.

A

Rotation already in place in case, you wondered the way you rotate a service.

A

Account token and turbine eddie's is by just deleting the secret that contains.

A

The service account token, then kubernetes will just create a new server.

A

A secret and it's rotated and another one which was actually just merged like.

A

Last week and released yesterday no connection whatsoever to this meetup, we.

A

Actually needed this, this feature, so there you go and then authorization.

A

Actually happens, that's the real feature that was implemented here. So there's a.

A

More complicated flow that that uses what's called the subject, access review.

A

Api in kubernetes, so how does that like work in detail, so I really taken the.

A

Example of prometheus accessing an exporter that is called coop state.

A

Metrics in this case, which is also a common component, I maintain basically.

A

Coop state metrics exposes metrics about kubernetes object, so, for example,.

A

Kubernetes has the deployment object right so and what coop state metrics.

A

Does is it takes the expected number of replicas and actually available number.

A

Of replicas, that's all in the kubernetes object right and it converts that into.

A

Metrics and that's really powerful, because we can then take those metrics.

A

And alert on them when the available replicas are not what we expect right.

A

But this is also super sensitive information because it most likely.

A

Contains names or stuff like that from our workload, so we want to make sure.

A

That that's protected, and so that's why we want to put the coop barbic proxy in.

A

Front of that, because coop state metrics itself knows nothing about.

A

Authentication and authorization, but our back proxy can do that generically.

A

So when we set up all of this, this is really the flow that everything goes.

A

Through so prometheus has a service account token that it has some role.

A

Bound to that, it is allowed to access any url. That is just slash, metrics and.

A

Prometheus does the request and the kubek proxy and the cube state metrics.

A

Processes are both in a single container in a single pod, two containers and the.

A

Entry point is the coop arabic proxy and at that point the coop iraq proxy. Only.

A

Validates that the token is actually a real token right and it uses what's.

A

Called the token review api for this, so that takes the token that prometheus.

A

Gave gave it the actual plaintext token and gives it to the kubernetes api. The.

A

Community's api then verifies that and returns the user information. So in this.

A

Case it would be a service account token, so it says this is the service account.

A

Token and you do whatever you want with that, and then the critical piece happens.

A

Which is it takes this user information and says? Is this entity allowed to.

A

Access this particular path, and it again does a request against the kubernetes.

A

Api which verifies that and then again returns the result for this and.

A

Obviously, whenever anything failed in this process, the barwick proxy returns.

A

With an with the appropriate error, but if all of this succeeds, it will just.

A

Proxy the request to coops eight metrics and that will return the actual metrics.

A

To prometheus, so really not all that complicated, because kubernetes already.

A

Exposes all of these api's? So what can this really look like? So there are a.

A

Couple of scenarios where this could even be interesting for like internal.

A

Services or something but there's some disclaimer, so don't get too excited.

A

Right away I'll demonstrate why there are also some issues, maybe around this.

A

But in general there's there are a lot of cool possibilities to at least secure things that are really tight to the cluster and.

A

I'll get to why, but something that we can, for example, do is that we can say.

A

User x can proxy to service y right and that's pretty cool because suddenly.

A

I can give michael access to access the prometheus front-end yeah yeah, but you.

A

See you see where I'm going with this, and this is really a super simple.

A

Configuration file where you just say on any given request. This is the subject.

A

Access review that I want to do so in this case, I would check that any.

A

Requests that I'm receiving is validated that the requesting entity has a role.

A

Bound to it that can use the proxy sub resource on the service named parameters.

A

In the default namespace, that's a mouthful yeah, so this is really cool. If.

A

You want to use the normal are back roles where you actually specify objects.

A

And kubernetes, but it can also be used for just something. That's called non.

A

Resource url ting- and this is what we typically in prometheus world use this.

A

To protect the metrics and part, but there was also a really cool.

A

Contribution that was done by someone from the community against the cube.

A

Arabic proxy, where they added transparent http to support, because what.

A

You could suddenly, then do is use this for g rpc services and protect for.

A

Example, tiller, if you're using helm, so not everyone who just has network access.

A

Against tiller now has cluster admin, because taylor effectively needs cluster.

A

Admin, so let's look at this for real a bit this big enough or bigger, bigger.

A

Okay, big enough okay, so I can't even see my.

A

Window and team oxen, you are okay, so the very first thing that I want to start.

A

With is just a really simple one, where I start apart within my crew, bananas.

A

Cluster, that already has a roll bound against it, which I can show here. So I.

A

Have a roll binding, a cluster roll binding actually because all non.

A

Resource urls have to be can only be specified in a cluster. Your a cluster.

A

Roll, whereas things that are local to a namespace, can be done in normal rolls. So.

A

Yeah we can see here that the default service account and the default.

A

Namespace has this metric standpoint roll bound to it and that roll just to.

A

Demonstrate what that looks like it's literally it just says the non.

A

Resource url is the slash metrics endpoint and an entity that has this.

A

Role bound to it can perform the get verb on it. So.

A

So I have this curl command, but I want to start with something else, so this is.

A

The one that should succeed, but I want to start with the non happy case for.

A

Example, so here I just requested just slash right, or can I subdue slash.

A

Kinfolk and I get forbidden right, but since this pod is running in the default.

A

Namespace and just uses the default service account which, by the way, never.

A

Bind anything to the default service account in the default namespace very.

A

Dangerous, but this should now succeed right. So if I request slash metrics.

A

Instead, I do actually get the metrics output from some sample application that.

A

I put in here so cool this works right. That's really.

A

A

But we saw there are also a couple of other authentication mechanisms in there.

A

So I wanted to show one more or actually two more so I have dex as an entity.

A

Provider, I think dex was mentioned earlier.

A

Awesome piece of software and dex is just a basically a federated identity.

A

Provider for o'reilly see why dc is a standard on top of oauth 2, so actually.

A

Standardizes the standard that was created to standardize something yeah.

A

But this is, this is really cool, because now we can use something that users can.

A

Use so, for example, as I gave the example that michael wants to access prometheus.

A

I don't have to give him a client certificate that is valid for forever. I.

A

Can just add, michael, it's email, so obviously michaels email is admin at has.

A

Example.Com and can log-in and we get forbidden. Why did we get forbidden now.

A

Who paid attention right so we actually requested slash, not slash metrics, so.

A

Good thing is again: everything works, hopefully works as intended. So if we now.

A

Request, slash metrics. We should again see our metrics and we do great so now.

A

The very last thing that I want to show is something that was actually just.

A

Released yesterday, which is a rewrite feature, so what we saw earlier, where we.

A

Only have a static configuration of which subject access review is issued.

A

When the coop arabic proxy is requested, the rewrite feature actually allows you.

A

To make this somewhat dynamic so that you can say, for example, if I have a.

A

Query parameter of namespace then use that namespace query parameter in the.

A

Subject: access review request and that way, for example- and this is what we use.

A

It for you can use this to protect and limit the amount of metrics. Someone can.

A

See in prometheus, so I'll show I'll show a small architecture about this later.

A

But I just wanted to show that this actually works.

A

No, I need to make this smaller, because there.

A

Delete that part that we used and run another one. So just a very quick.

A

Overview but we'll talk about it, talk through it a bit slower again, the way.

A

This works is there's another proxy in between now so there's the cooper back.

A

Proxy, which rewrites based on the query, parameter, name namespace the subject.

A

Access review that we do for this, so then, additionally, when I do a request.

A

Against prometheus in this case- and we can see this this here- I'm using the.

A

Query: api of prometheus to query the metric called container memory usage.

A

Bytes- and I want to do this for everything or I'm just requesting this.

A

For everything, right and typically prometheus would now give you the.

A

Container memory, usage of all containers in your cluster, which is most likely not.

A

What you want in, like a multi-tenant environment, for example, because then.

A

Team, a can or company a can read metrics from company b and that's obviously not.

A

What you want so we built this thing and it returns data. So that's that's great.

A

Right but the more important thing is, I only.

A

Have role bound to me that allows me to access metrics from the default.

A

Namespace and now, which namespace what we really wanted to see metrics from in.

A

Communities yeah, but we can't yes, it works yeah. So let's go back to the.

A

Presentation to see a bit more about that or yeah we'll see about bit more.

A

About that layer, but there's one fundamental flaw about this and if.

A

You've ever actually liked, if you've dealt with tokens token authentication.

A

Before you probably know about this problem, which is tokens, are just plain.

A

Text that meteors gifts to some entity proxy in.

A

This case and the proxy has the plaintext token, so the proxy can take.

A

This token and pretend it's prometheus and potentially escalate.

A

It's the privileges, it has right, that's not a good thing and that's. Why really.

A

In the current state, this is in only secure things that already have higher.

A

Privilege of things that are going to request it does that make sense so and.

A

That's because or that that's because otherwise it would be a privilege.

A

Escalation, it was it's actually the other way around it. What I said the.

A

People that request have to have an no sorry. I was right the first time.

A

Yeah so, but this is a general problem in kubernetes, because the exact same thing.

A

Happens with the couplet and sometimes even the api server and the couplet are.

A

Configured to do this, this dance- and this is not what we run right, because if.

A

One couplet is an attacker, get gains access to one couplet, it would have.

A

Access to everything immediately, that's not something we want so there's.

A

Something that's called the token request: api that's being implemented.

A

Right now, where it can say, I want an additional token, but only for this.

A

Audience so in this case, what we would do is we request a new token only to.

A

Request, metrics from koop state, metrics I'll be done soon, yeah and then.

A

The question was: would I not request this from the identity provider? What what do you mean by this.

A

Yeah, this is exactly what happens so prometheus goes to the kubernetes api.

A

And requests a new token only for coupe state, metrics and then coupe site.

A

Metrics again, validates was this token only for this audience, but technically.

A

That that's only it's not strictly depends. So the question was texts is.

A

That back snap, the identity provider, in this case kubernetes, gives the identity.

A

Because it's a service account token right, dex could be an additional.

A

Identity provider that you also give identities of dex access to the.

A

Kubernetes cluster good question, but yeah and this way we can make sure that.

A

Tokens can only be used for the thing that they're meant to be used and not.

A

That the proxy can go ahead and ask the kubernetes api had given me all this.

A

Nice stuff that prometheus is allowed to to request because the criminal side, api.

A

Then says no, this was meant for to be requested for you so yeah. This is.

A

Basically, kind of reinventing kerberos yeah, so I'm already mentioned the.

A

Rewrite functionality and really the it's super simple: how to use this? You.

A

Just add an additional entry in the config file and then you can use go.

A

Templates to configure this and most important about this for this to.

A

Actually be secure is that when this is configured and no query parameter is.

A

Given that's a bad request, because otherwise we would just let this through.

A

And give access to everything again so thought about that so yeah really what.

A

This then creates is something that we kind of named soft tendency for, for.

A

Prometheus for the prometheus api and for for the federation and find, if.

A

You're familiar with that yeah, but this is also only safe. If users cannot.

A

Configure what's being scraped because otherwise you can pollute the tenant or other tenants right because in prometheus.

A

Labels are freeform, so someone could add additional labels and pollute other.

A

People's data, so this is very specific use case if this is what you want to do,.

A

Or if you want to use this, come talk to us and we'll see if this actually makes.

A

Sense yeah, so some future work that we want to do so. The token request api is.

A

Not actually merged yet in kubernetes, but as soon as that is merged. The second.

A

Entry is what we we will use to request to validate the audience on every.

A

Request and something that we've also talked about is right now the group are.

A

Back proxy itself needs privileges. To do the token token reviews and token.

A

Subject: access reviews and it would be nice if there could be some mechanisms.

A

Where the koopa park see wouldn't need anything right kind of like if we use a.

A

Pki and say we just whitelist the entities that can access this because.

A

Most likely prometheus might be the only thing that ever accesses that, but again.

A

This is the kind of use guys where we need to see if that makes sense,.

A

Yep, okay, thank you. You.