Kubernetes AWS Provider, 6 Sep 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: kubernetes sig-aws 20190906

Description

kubernetes sig-aws meeting 20190906

A

Hello, everybody: it is Friday September 6th 2019. This is the AWS bi-weekly meeting criminals addressed by weekly meeting I. Am your moderator, facilitator, Justin, Santa, Barbara I work at Google, a reminder that this meeting is being recorded and will be put on the internet. Shorts be, and so please be mindful of our code of conduct and be a good person. I have pasted a link in the chat to the agenda.

A

We have only a couple of items on there, but please do put your name in the attending list and if you have any items you would like to talk about, please add them to the agenda. Otherwise we can hand over to our first for exciting agenda item which is Micah. You have a demo for us, I.

B

Do so we're almost there.

A

Time we can go out of order if you'd prefer yeah.

C

Nick, if you want to go first yeah minds I, just it's a very quick update, I was just gonna say that yesterday merged actually I guess. Yesterday we merged the oh one to release of the at mesh controller and it adds initial cloud map support. So you can basically, your your pod. Endpoints can be registered with AWS cloud map, which is a type of service discovery and app mesh uses cloud map to basically tell the envoys that it manages about those endpoints. So, instead of using cube dinosaur coordinates as your service discovery.

C

You can use AWS cloud map, which is pretty cool, so that's it. That's it's merged. I think we still need to add some documentation on how to use the cloud map support to the repo, but other than that you can start using it right now. That's.

A

Awesome thanks: do you want to give a 30-second description of what app mesh does yeah.

C

Absolutely so at mesh is a AWS service and it's sort of I think of it as sort of an envoy config generator.

C

So it's a fully managed service that generates envoy config envoys are l7 proxies, so you can run them as side carves to your containers and those you know if you're not familiar with envoy, it gives you basically metrics on your applications, so you get some observability and client side, load, balancing and mutual TLS and stuff like that, so that you have a little more control over kind of observability and operations of the applications, and so at mesh.

C

Is this service that you create resources in at mesh and those resources are translated into envoy config for you. So you declare your backends for each service at mesh will generate the config, send it down and then your services, your applications, can talk to each other makes sense. That's.

A

Layer, seven inside the cluster and does it do like layer, seven ingress or is that where we go to alb and things like that,.

C

I think you can configure ingress, but it's I think right now we don't really support it. So I would say you would go to alb for now, but that's something in the future.

A

Awesome, thank you for the description and yeah. That's really helpful and exciting zero one, two uh mica over that we had the floor to.

B

You I am ready. Okay, let's share my screen, really quick, see that shares the rights me. Okay, yeah.

B

All right, you can see my screen.

A

Yes, the text is readable, but pro yeah don't be bigger for the elevating yeah. Let.

B

Me make that a little bit bigger for you. So what I've got here for the demo is just a really really simple example.

B

So I've got a config map, I'm gonna mount into my pod, just just as a example, but I'm mounting one thing just for a second part of the example, which is an AWS config file, but also this primary script is: what's gonna get running the pod, so the script is just installing the AWS CLI, echoing out a projected identity, token and then calling get caller identity. So to give just a little bit of background on what will be going on here.

B

The feature that we just launched this week for eks and for even nani casts just AWS clusters or even non AWS clusters. You can actually run this in a Google, a DCP, cluster or GK cluster, or just about anywhere. You want that where the cluster has a public identity doc.

B

So what this does? Is you run this Web book in your cluster for new eks clusters? This comes built-in. In the eks case, you can describe your cluster and there's a new field on the cluster description called the oid c discovery, and this endpoint is a URL that contains.

B

Oh, I DC metadata and one of the key pieces is the signing keys for projected service account contains kubernetes has two types of service account opens. There's the legacy service account tokens which you're probably familiar with this is what gets autumn out in every pot. It's for you cause to talk to the API server and authenticate that way. Those are static, tokens they don't expire, you can delete them and that the service account token controller, recreates them, but it's really difficult to might migrate. Rotate.

B

The signing key for those you'd have to at a new new signing key delete, all the existing tokens, which could be really disruptive and then get all new tokens issued with anything. It's a long process that you don't. Nobody really ever goes through. The news type of service account tokens. The projected service account tokens are temporary and they're. Actually, oh I be C value. T these JSON web tokens. They support configurable audience so legacy service. Account tokens can only talk to the kubernetes api server.

B

These projected service account tokens can talk to the api server, but they can also be used to talk to other systems and in this example, we're using them to talk to AWS.

B

Aws has had since, like 2014 this concept of I get external identity providers, so you can have your either sam'l or o ID see a provider allow users to authenticate AWS without having to create an iamb user with password for every user, so you could make it a Google octa whatever, but it senses, just oh IDC and kubernetes supports, give doling out o ID c tokens. We can put these two things together, so in this example, I have a kubernetes cluster that I just created yesterday, and this is the token issue or end point.

B

So if we look at this, it's actually really quick, known, open, ID HN. We can see the open, ID config. Oh it's not working.

B

Yes, there we go and if we go to the keys endpoint, we can see here's a site, it's a RSA signing key. So this is just for STS, Amazon, secure token service or any other provider, or any consumer of these tokens to see is when I get a token. Is it properly signed embedded in the token it's this URL to say, go. Look it up here and see if this is actually signed by the cranky.

B

So that's kind of by way of background. That's just just to let you know kind of. What's going on in the back, so kubernetes is now issuing these token saying, look here for validating them and the key configuration kubernetes is, you can add a new volume type called predicted volume token, and it mounts one of these tokens into the pot. Now there's a lot of there can be a lot of configuration to set that up.

B

So what we've done to make that a little bit easier is added a new web hook to our eks clusters, and you can run the slip up in your own cluster. What it does is when a service count is annotated with a with an AWS roll arm like this, and you given up given a certain pod like this, every pod create will be evaluated to say: what's the service account on it? Look the the web book will look up. The service account look for an annotation.

B

If the annotation is present, it will add the following environment variables, AWS roll iron and token identity file. So you can see those two here and then also a volume mount so creating a print. This whole configuration of a projected bond up. You don't have to use the web hook. If you don't want to and the annotation you can add the environment variables and volumes to your Web book. If you don't want to use the book at all, that's that's fine.

B

The web hook will short-circuit if you've already added them and you've annotated the pot, and so there won't be any adverse reaction going on, but that's that's basically what I'm going to demo. So the other point is every AWS SDK at this point has been updated with support for these two new environment variables. So if the SDK detects icy, they do its role, Arn and web identity token file. Instead of going to the node level instance metadata I'm going to call STS assume role with web identity.

B

It's an existing API call to get my credentials instead, yeah instead of doing you know: I am bs environment variables, AWS, config file, etc. So, let's, let's see how that works. So back to the demo, I've got a script. I'm gonna run, install the AWS PI echo out or cut out the the current token file and get called in and I've I'm gonna create a service account token, with this role, Arn and one other key pieces.

B

I have there's two directions in the trust here on the service account I have to say which role I want to assume, but also on the roll itself. I've edited the trust relationship to say, I, trust this issuer and I'm gonna actually lock that down even further to say that it needs to be for a specific service account, so I go back actually conditioned had this condition policy to my I am trust document. Let's remove that guy, don't need this I didn't used to subject. I, don't need a scope in the name space.

B

B

And I'm going to add this really quick: this is a live demo.

B

And this my service account is what we called over there. Aws say: AWS IRS a demo, so we're saying that this role can only be assumed by this specific service account if I didn't add the string equals than the entire any service account in the entire cluster could assume the role if I wanted that so I've tightened that down now. So, if we go back to this and then the the last thing is we're creating a deployment, we're just have one: it's gonna run the script on entry.

B

It's the Python, 3 image, really simple and we're mounting that script as if in Sigma, so can you all see my screen is that is that big enough now, before.

A

Your black, my good, do you want to not print the token token? Okay, it's uh it's good point.

B

A

A

B

Oh, that's probably good call I like to it. Oh I'll! Stop that I usually do that just for inspection, because it's kind of fun but we'll remove that so we'll get the just get color identity all right. So.

B

The token is revoked as soon as the pot it's bound to is no longer active, so okay, but yeah we'll go ahead and remove it in okay, excess. That's only on a to the kubernetes idea, best assistant now, alright. So we're gonna create this really quick.

B

So this will create the config map the service account in the deployment. So now we have all pre created.

B

Let's get pods. Is that pod ready? Yet.

B

Okay, it's running and.

B

I'll get the logs out of it and we'll see that the identity in there- okay, it's still installing it- was clie, really quick, predictors kind of token we're, not I figure it out. Okay access denied, that's probably the string equals I just tested this. Without that I probably worked this condition really quick. So let's remove the condition that.

B

It's going to restart and we can actually just follow that again.

B

Or might have to delete the pod I will just delete the pod.

B

A

A new one it looked like there was a cool trick in your Jason. Where you have a document comment in there. I don't know. Is that valid? It's.

B

Just a hat on JSON that, if you I don't know if I haven't actually tested to see, if, if SCS ignores so stop yeah.

A

D

What I was wondering what.

B

They're, like that, would be a great feature, a lot of like go. Marshall go and Marshall will.

B

The goat, JSON library will, by default, ignore fields. It doesn't recognize so forgo it works really well, but for not every other system, it necessarily works. Let's try this again.

B

So this should be installing the claw and then there it is okay. So what we have is the role that we wanted. So that's the s3 reader role and then the the database flies using photo core underneath and it generates a automatic session me if I were to actually do this again. uh I go to my deployment, channel and I were to uncomment.

B

This annotation will see that that's actually a different role, so.

B

It'll be the underlying rule.

B

And we'll need to delete that pod.

B

Really all pods, because there's only one.

B

B

All pods again.

B

It's her new pod will trail that.

B

You okay! So now you can see it's using the note in since well it's using the instance metadata the instance metadata is still on by default in the UK Asami. It's up to you to sort of block. My MDS instance metadata access from your pods, because during a migration period, if you're using the node role for everything, that's up to you to determine where that would come in. But that's that's kind of the demo. I can show one more thing: hey.

E

Myka I'm not sure if you address this earlier but I I- note that you are executing this demo on a 114 eks cluster. There isn't anything specific about the webhook that that requires 114 is it? Is there.

B

There there is not it's the web ago, actually I think works with even kubernetes 112 like upstream kubernetes 112 for non eks. Docs are forthcoming, I'll, be adding that to the web book repository Beach, if you're, not if you're, using 112 or even want 13 or 14 a.

B

Rsa key that you, a second RSA key you'll, have to give the API server private key and public heat as it uses the private key to create the tokens. These projected service account tokens. One of the key differences is they're not stored in a TV like legacy service account opens or you can get them as secrets. So not only are they short lived and, and they automatically rotate, so the cubelet will remount a new token before it's expired in the pod, but the yeah they never hit in the database. So they're ephemeral, all right.

B

B

Create this recreate this pod one more time, it's one more thing: I want to show, so this works great for directly assuming a role but a lot of users who are using cube, Diane, ki and etc, want to do chain the zoom roles, so I have a role in one account or a second role in the same account or a role in a second account and I want to actually use that other account role within my pot and that's totally possible via like for the two primary ways.

B

The first is just how I created an OID C provider, in my account that the cluster is in with this URL, this URL I can use and that's a per cluster URL I can easily create that same identity provider in my other account and have my pod, the annotation on that pod, directly assume a role in a second account. There's nothing specific in my cluster, that's finding the pods necessarily to my to my account right or the the service account provider. So that's one way.

B

The other way is you can do a chain to assume role. So that's what I just want to show really quick. So if we get if we get pods again so we'll have a pod, that's now annotated. It has the token and everything okay. So if we go in here.

B

Shell into the pod really quick.

B

But I've already have a config file, AWS config file, and it's got two roles in it. It's got I'm sorry. The first role is the one that I want to assume with this web identity token. The second is another role. It happens to be in my account, but it could be in any other account in this second role. So two things the first role. I'll share. This really quick has two permissions in list buckets, and it has a second permission that it can assume this second role, don't make the SDSU role in the second role.

B

I. Go to that. Second role, really quick, go back to roles you can highlight and show how this works. So the second role has a trust relationship that says my account. Id can assume it. It doesn't have any permissions by default. It's just a demo that the role can be assumed. So if I unset either US role, Arn.

B

All right so there's no token file here. Oh, this is not good hold on.

B

Oh, it didn't get rounded I. Think I'd miss the annotation all right. Well, that's how it works. I won't keep everyone waiting. I know it does work, and if you want to wait one more second I'm sure we can get this working I.

A

Think this is really great: okay, god. This is the complexity of chaining rules. I, think it's it's great to have fun identity in there. I think.

A

Yes, the a lot of people who develop their own solutions, and this, like the this, feels like a more secure option because of that additional annotation on the I am side I inside.

B

B

Will get ya know we want to see that the volume is there. So we'll just double check that that volume is there again, we've got our new pod and you can see okay, how its projected volume token get the pod name.

B

That's the label of the names. What we want.

C

B

How did I not move copy.

B

A

We don't need to put your this home I'm sure you could like put a video up of this I'm like link it to the repo yeah. It's.

B

The whatnot right it is, it is it's the so I wanted to just demo the role, but you know what we can do. Yeah we'll have to do it. Another time I have I, have not thoroughly prepared this demo. It was tried to just show it quickly, but we'll save everyone's time. For that. The.

A

First, there was also and I think yeah having the yeah. It was I'm sure it works. It's just yes, it's it's great to see. Everybody wants any questions or anything like that that Formica, it's just too busy downloading it to try it I have.

D

A quick question: this is my dobby um you'd, veteran cubed, I am and different migration paths. Is this compatible full where it can run at the same time is cubed. I am is if we were to be switching from cubed I am over to this method, or would they each other I haven't.

B

Tested it I, I'm, not sure I, think cubed I am in. If Cuba I am impersonates like the instance metadata, then this should work fine. Actually, this should just most SDK for SDKs the web identity. Assume role of identity flow is before the instance metadata. What you saw here, like the instance metadata, was still enabled and if I, if the token file in my River wasn't present, it fell back to instance metadata. So this should just take precedence over that and it should be a fairly easy migration.

D

That sounds great. This is really cool, I'm, very impressed and happy with what I've seen awesome. Okay, we have.

B

B

Okay, so we're getting the first profile, get color identity, which is what we expect s3 meter now we'll try a second.

B

There we go so you can see it used a first role to assume the first role and then using that credential assume the second role. So that's, that's all I wanted to.

A

That is really great and yes, the the point about the compatibility is also an awesome observation, but this is yeah. You can get there one part at a time or when deployment inside so.

E

Micah, besides some of the documentation things that you mentioned earlier, is there anything on the sort of roadmap for changes or enhancements to what you demos here today. Anything upcoming that you'd either like contributions from the community on or feedback I.

B

Think at this point, we're kind of just like excited to see people using it and see what what new needs people have. I think this will solve like a lot of a lot of use cases I think as as those needs develop like I love, I, love issues on the the github, repo and and contributions as well like I, want this to work really well on, like all cloud providers, so that, if you're running this anywhere you could you can use this web hook to get a get a token and and talk to AWS.

E

Awesome yeah I have now for to the repo and we'll you'll start seeing some some PRS from me with Doc updates and stuff awesome.

A

Very impressive, thank you. Mike I, don't know if there's anything else which anyone has added to the agenda, because I think we have yeah. We have reached the end of our agenda. I, don't anyone else does anything they would like to bring up. Otherwise, everyone gets 30 minutes back.

A

Going once going twice all right, thank you, everybody! Thank you. Mike yeah.

B

Awesome work, my gun for.

A

The update and see everybody in two weeks see you right right.