Istio IstioCon 2021, 10 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Know your peers

Description

#IstioCon2021

Presented at IstioCon 2021 by Alex Van Boxel.

It’s lonely in your pod, but finally, you receive that long-awaited knock on the port… but can you trust that inbound request?

This session dives into an essential aspect of a service mesh: trust. We’ll dive into how certificates work into Istio, use peer authentication, and explain concepts like SPIFFE identifiers. Peer thrust can also be leveraged in the application architecture.

A mesh is not only for cluster administrators but also for architects and developers, making it well worth to highlight those patterns.

A

So, as pedro already said, I'm alex- I don't do slides about me myself anymore, but what you kind of need to know is that I'm now kind of involved with my third easter uh deployment, uh all three of us with another in intake the first one was more on observability.

A

The second one was more about routing to different kind of machine learning models, and this one, as you may be guessed, is more uh tailor to work like identity. So that's why we're gonna? That's what we're gonna talk about, but uh a disclaimer.

A

So there is a lot like a lot of text on these slides and that I know that is against the rules of presentations, but um it will make the slides more uh usable uh afterwards. So don't look too much to this to the text. The the keywords are highlighted and for the rest, you don't have to bother I'll, probably tell you what's on the slides.

A

So, what's the goal of this talk, so it is actually to gain some deeper understanding on identity within the istio ecosystem um and having that knowledge will hopefully help you, or at least developers that will deploy on these networks, help them design their architecture. Knowing that they're a lot of things that they can actually rip out of their codes and just rely on so and what is it not? It is not. um It doesn't talk about any of the hottest new features.

A

uh All the things I'm gonna tell you is uh is already there from day one. um It is actually just about the underlying plumbing, and it's actually a talk that I would have hoped that I've got before, starting with the istio, but there were no talks at that time. So because the first time I was involved with this still it was version 0.3.

A

um So it is actually to make sure um to overcome your fears right, not knowing what uh that this makes everything kind of magic, and there is a long quote that I'm not going to quote of an astronaut not is described in a book. That's when you do not know any of the concepts. Everything is scared, um and actually um this this is scary.

A

If you see it like the first time, if you turn it around, it even looks like a rocket so and all those astonished they they like to understand a bit what's in the rocket right, so they feel comfort and comfortable working with it. So here this is the purpose of this talk as well. um If you want to talk about identity um in istia, there's no way around spiffy spiffy is is a cloud native identity, spec.

A

It's it's pretty new, it's it's a project! Well, pretty new! um It's a project from the cncf just like kubernetes and so on.

A

um There are already kind of a lot of other products uh using it and it has also its own own implementation. But the spf is it's: it's a spec so full out it's secure production, identity framework for everybody, so I need to google it every time because nobody really remembers it. Everybody knows it's just like like spfis.

A

uh If you talk about spiffy or actually identity in the clouds, um it's it's kind of tricky. um So you could talk about kind of the platform um maybe or on aws, and you have some machines. All those machines have a unique identifier.

A

um You could talk about that in within your your network, every 5, 000 machines. Everyone has a new, unique, unique id, so you could take that as as an identifier, it's kind of tricky when you go cross-cloud. So if you're on google then and aws and azure, you have like you're multi-cloud how they're they're not really compatible they're, they all look different.

A

They have another way of getting that identifier, so so that's tricky.

A

Of course, on all those vms and notes, or whatever you have like the hosts like the real vm, its operating system, it gets more concrete um in general, it has a host name, an ip address, so you could start talking about that. It's more generic!

A

um It works across clouds right, there's, an app is an ip, um but not every ip is directly accessible. Sometimes you have like nets or translation between ip addresses. If you're, a really big network, um then again it it, it can gets, gets, gets tricky and the more and more you cloud native you go um the more and more you're. Also talking about hosts going away, you're talking actually about processes right on kubernetes.

A

um The way you you want. You want to cast a lot of processes on on on your notes, bend back everything and those nodes go away, so the processors can go to another hosts. So the only thing what you actually care about when talking about an identity is how do we identify that process, not that process running on that host or not platform? No that's that process.

A

We need something stable, stable so, and that is what spiffy actually tries to to solve is how do we get?

A

How do we give those processes, a stable name.

A

And, first of all, we should kind of agree on how what do we want to call it so um and that's why part of the spec is the spiffy identifier, so it is actually the core of it. So what is the name of the process?

A

And it's pretty simple? um It just says it's a uniform uniform resource identifier uri. It really looks a lot like your url that you type in your browser, although not a lot of people, do that anymore. They just google it, but it's like um http host name, slash part um speaker just looks like that, so it starts with its its beginning, spiffy and then slash so that identifies that this uri is the spf identifier and what you have in http as a host name. There.

A

You have your trust domain, so that part doesn't identify any machines. No, it is kind of a domain. It's like scoped every every host, no every name within that trust domain is kind of trusted. So it's it's handed out by a certain a certain authority, and after that you, you have a part. So this is actually a valid spf identifier.

A

um It's not really human readable, um but uh but it's valid, so you could have like some system. That's that identifies uh processes by uh you know uh uuid, for example, and then you would have something like that.

A

um You could go here right here, um so saying: okay, we have like a payment system and that one has its postgres database, uh another component, something like that. You could have a mechanism that that creates those those spots like that, like grouped per service for you, for example, payments uh postgres, um your shopping, cart, uh brush class, something else, and so, but but actually in the istio world. um They've chosen uh a more um a more stable kind of uh system.

A

um This year is mainly deployed on kubernetes, so it hooks a bit into that system where you have the path identified as ns, and you guessed it it's the namespace and then slash sh chase and that's the service account.

A

It's it's an important thing to remember that the incubernatus, the is the service account that is used to identify your process. So a good weight is a good way to to freeze to to get stable identifiers. Is that you give each kind of service a stable, its own service account already talked about the truster bain right, that's the first part.

A

um Mainly you will see it. um If you don't configure anything, it will be clustered local and, and that is done within your cluster.

A

Those work, those processes identify each other each other. But if you grow your network go multi-mesh or something you can choose to have each cluster, for example, uh another trust domain.

A

So you could have like production staging or you have you can you have have your company name um in there and that's kind of up to you and that really will depend from business to business so but it's kind of nice to have like a convenient way to to name your processes, but uh can we trust and identify well for for having that trust or or making sure that we are talking about this process?

A

um The the second part of the spec is uh talking about the s fit is a spiffy verifiable identifier document like the they have like a lot of long names in in that spec, but, as fits for short, everybody talks about as fit, and it is actually a way to to make sure that um that the identifier that we talk about is trusted.

A

And how do we do that? There are two two parts in spec, um two, two different implementations right. So it's it's kind of using spiffy using a lot of things that already kind of exist. So it's it uses or the swt token, so a lot of um web developers and so really like it just put that in your authorization. It's just a json object, putting an authorization header and you can decode it. There's certain rules.

A

um Istio doesn't do this out of the box and and later on in the stock. We will see why that I would reserve the jwt token for something else. So the the current uh implementation in istio uses x509 uh because it uses the uses mtls.

A

So the spec has has three parts, um but I'm going to talk about the workload api a bit later on, because that's kind of deeper it lies in deeper into uh deeper into history.

A

So I'm going to go now to the deep redo x, 509, but be sure to check out the spf spec and it's kind of sister project spire, there's an implementation, more tailored to words vms, and they actually work because they they have known their own spiffy identifier, but it works fine. They talk to each other because they they kind of know how to search who's who so x509.

A

So it's kind of an old technology.

A

It's been updated a few times, but it was first published in like like 32 years ago and it's actually still still valid uh valid technology for the youngsters around here, probably it's older than some of you, um but it's still used everywhere, so your browser um and for for securing networks, um people that come into contact uh with those low-level kind of certificates. This. This is common way to distribute your your your certificates, which you just spend them encoded binary plop.

A

And if you use my favorite uh tool for decoding that so open, open ssl, um you can actually decode what's in there. So it's it's a pilot blob. um You see like a lot of attributes that actually identifies.

A

A lot of things, what is the name or the signature an important part there is the public key and we're going to dive into that directly and the signature so almost well. Every certificate has a public key and it is signed by and generally somebody else, and it has some interesting attributes and like like here. If you, if you look deeply, you will see a spf identifier.

A

Talking about certificates right so to the right that is kind of each workload will have a will, have a certificate and it will be signed by something that istio knows and, and that has its own certificates, um that certificate always comes kind of the one that owns it always has the private key. That is not part of the certificate that lives outside it. So you so the one that owns the certificate always has boats. The certificate is public. Everybody can can have that certificate, because it's even pasta but passed around that public key.

A

That's the only thing that needs to be seen so to react. Your workload will have that certificate um that identifies that the name um the name of the certificate it will have that uh that spiffy identifier in that x509 data and it will have the public key and istio- will actually sign it and we'll go deeper in how it will sign it with its private key and everybody in the network or in the trust that can can can look up that trust uh by just verifying that signature with the public.

A

Key that like this, um you can have several levels deep. Certainly, if you're planning to go multi, multi, multi mesh or have like uh want to uh communicate with spire or other things, you will probably have like one root certificate.

A

uh Your easter and your cluster will have an intermediate, that's the middle part and- and you yourself will learn the certificates and you can nicely see who is signing the private key he's signing the one in the middle and the one in the middle. The intermediate will sign yours and, in the other way around, you can verify everything to the root. So you can verify the signature by the public. Key of your intermediate and intermediates can be checked with the public one from the russet one special one at self science.

A

A lot of people probably heard about self signing. It is just the root. The certificate is always self-signed. It signs itself with its private key. So you can actually verify that your self-signed by the thing that you're verifying your signature with a public key in the same section.

A

um In a previous slides, I didn't show kind of the private key of your own one, because in in the signing process or the building of the trust that private key is actually not not really not really used, certainly not passed around.

A

So with that in that certificate you will see, then your your spiffy identifier, you so already kind of saw so as a peak and and so if you did diver it's just a uri and that's part of the subject alternative name.

A

So there are two parts in your network, so the the signing certificates or the intermediate. There are some rules uh that spiffy says like your you. Your signing certificate may not be used for for a workload, um it can itself be a spf identifier, but you still cannot assign it. In generally, it's like just the speaker, identifier with the trust domain and there's no part component and the leaf once for your workloads.

A

It has a full spf identifier that that identifies you.

A

A

That is part of how how workloads in the network build trust between the client and the server right.

A

So I want to go through the through the messages, how everything is kind of built up, um because this is how two workloads- interchange, um those certificates- and it's pretty interesting to know- and I come back after it is something you you need to be kind of aware of is kind of what the implications are right. So so your clients, you want to start kind of a secured connection, so you start with a client here, uh client, hello, you say: oh, these are the ciphers uh that I support.

A

uh These are the completion algorithm, here's, some random stuff and the server, so each block is kind of a message. Server will respond with with a lot of other messages. um It's just saying with the server hello, okay, um this is the cypher. I'm gonna pick I'm in control. This is a compression. I know that you understand this. This is what's uh what we're gonna pick and here's some random randoms uh crypto stuff again, so the server is always in control on the cipher.

A

It will also send it's um it's public key. Like we said, there's no no going around with um with private keys, it's it's just saying: here's my public key and you have the rest of the chain till the route.

A

So it's intermediate certificate. It will also send um some some server exchange things. That's not important in this story, um but in empty ls, because we're kind of we're we want to trust the client right, we'll both be directional trusting. um It will ask um for the the client's certificates in this message. So that's not what not happening when you're on the web, when you're just browsing only the server kind of tells the client here. That's my here's. My thing, that's, who I am um in mtls here.

A

The server wants to know who the client is, and that happens here right. So then, at the client you it's it's client's, turn right it will. It will verify that the signatures, uh all the 509 bits are, are correct right, so it it has the root certificate. So you can check all those things so it can can build the trusts uh but yeah. Then it says, like I heard, an annoying server wants to know me. Okay, I'll prepare my certificates, I'm gonna send it to the server at this moment.

A

um An interesting thing is, though, and where the private key- and this is where the client really can prove that that's him, uh because it will use the private key to sign all the previous magic messages with the signature and the server can uncheck check that with the public key, so so almost finished, so we're wrapping up the server rule will conclude we'll check that everything is verified signed correctly, and it will also will do so by following the chain um so and that's like like just following a chain from the route on the right, because it knows the route the clients sends all the intermediates.

A

So it has all the information to build the trust. Once that's done, it's kind of, then you have to use your channel and kind of the. The client can send its bits and bytes over the channel um bi-directional right and and then then it's it's just like business as usual.

A

um You can. You can see that kind of it's kind of a ping pong, certainly in the beginning. So this is why well worth to know that it's good, that you keep connections open right and that you work like if you you want to have your ssl pipe open and do not do that for like if you want to send five bits like star start start all over, because all the handshaking goes uh over and over and it will make it very slow for five bits. It's like more.

A

I don't know how much, but if I can have a lot of data, send over left or left or right so voila. So so that's that's kind of important. So after the the tls handshake, you have your pipe, but also the both parties know who's who.

A

An important building block so after kind of all the ssl parts, um uh because we've still talked about kind of implementation, who's doing it who's doing actually the artwork so and that's the envoy, so within istio envoy, is actually the the real, um the real worker. So if you have like easter as an end call on it and then an envoy would be really the work, so the the typical, if you see the typical uh istio uh diagram right, it's everybody will should have slides like this.

A

Like this, you will see envoy and invoice, so I've left out the control plane because we're going to talk about that later on, but every workload has its own envoy. You have like ingress and egress, maybe even multiple ones. That's also just an envoy, and always it's all small blue box, and that's kind of the this, your agent that is running next to it.

A

Interesting to know is that, and I doubt that it's kind of used a lot, but I I think it's a very important thing to to dive into it's kind of um envoy can identify to the application running on top of it. What the client certificates are because remember envoy, is doing all that handshaking.

A

It's doing that on behalf of the application right, so your application doesn't even know that that is kind of tls or or secure channels going on. It doesn't know because it just does just lives in, from its view, lives in an unsecured view.

A

uh uh Oh worlds, so it just sends out plain data plain uh if you're talking http um just plain out, but it's very important that that you have that feedback cycle so envoy will send like a lot of headers to the application and the x4 client certificate is an important one because it just says what a certificate looks like from from the client. So these are the ones that it's, it's all part of the envoy uh documentation.

A

So those are are the blogs that are, that could be part of it. But what is kind of important is the by the hash, the subject and uri, so the buy is actually my own. If I'm the server.

A

um That's my own certificate uh to make sure that that I did kind of make the this um this or my proxies make this a header is a secure hash, the subject or the and the uri, and that is the uri from the client. So I actually can identify who the client was an application-wise. You can do some stuff with it, so in istio, how does that all those things come together? So you have the now one one big monolith: now they still demon with some some interesting components so bringing it all together.

A

um How is that trust built up because yeah your workload? You still need kind of a way to to build that trust up and that's kind of happens in the agent. So it makes like a key key value, a key, a private key and assign a certificate. The sign-in certificate is almost a certificate.

A

um It sends that to the the istio daemon, only the certificate signing requests.

A

It can actually verify um who it came from, because um the issue daemon uses the token of kubernetes to to to make sure hey. I'm I'm valid, uh I'm I'm really that person, so uh you studiement can check that it will create a certificate and it will send it back to the studio.

A

Then envoy is has several xds protocols and xds is discovery services and we're gonna use the secure discovery service to send from the institute demand to the envoy for a proxy and from that on, we just can do uh unsecured communication from the workloads and and and android proxy will will kind of encrypt it all for you and on the server side right it's.

A

It is exactly the same, but universe, and here the last part um is where, where you can then inject, where every will inject those headers and where the application is kind of aware can be aware of who the client is so what's kind of important here is that identity is established at startup um and that the control plane is only needed when, when there's a kind of a refresh, the tokens are pretty pretty short, but it's your demon can go down without actually the mesh going down because it just keeps on encrypting and validating, because you have those certificates, they have a certain lifespan as as long as those lifespans didn't kind of cross.

A

um It's all good.

A

How can you leverage that now in your application or architecture right, so it's kind of important to have like a difference between you to identify the difference between peer and request uh authentication, so the the peer authentication that we we talked about is everything.

A

What I said was peer authentication who is talking to me right and and that's how you can leverage, but then, on top of it you can identify request, authentication and if you can use them the authentication for for your requests so from your clients from if a client makes a request, it can identify itself and in general that will be kind of the human.

A

So looking at a non-mesh uh artery, that's how you could kind of build an application, so you have the you so so the job token from outside um comes in it identifies the human, um but a problem not to rush world is kind of that you need kind of. If you pass from the presentation layer to a payment service, uh you cannot use the job token from the user, because you want to actually establish kind of a trust between your presentation and the payment service. So you probably use a job to open. For that.

A

I don't know how do you pass along and the identity from frame recorder? You can do that in extra headers or something but there's not no real, real good standards so in the istio mesh world. Well, it looks like this kind of grand that it looks like a lot scarier, but as an engineer's view, it's only this it doesn't.

A

It can really use the job token, because you don't have to establish trust between your pres presentation layer and your payment service, because that's done so, I was saying kind of and yes, ingress right, so ingress right um that job token, so that jet token we're not handling it. In initial on how you get that jojo, okay, that's generally kind of application, specific or something, but it just passes through the trust is- is due due to mtls. Anyway, it goes to your presentation layer there.

A

Something can be done yes or no in the application um in general in application layer. Maybe not! Maybe, yes, um it's all up to the application.

A

um But then, if you go to the payment service like in the example right, you can just pass through the same jaw token you don't have to translate it anyway, anyhow, just pass it through. um It goes to the envoy proxy. The trust is between the presentation and the payment. Anyway, you do have, though, the possibility to have extra policies on the request or on the pier. So I have a quick example, the only crd that I have here, um so that's authorization policy that actually uses both.

A

So you can you can, if you, uh that is that's a snapshot from documentation, but if you you could select your payment service um and in the from section, that's that is using like the spf identifier. So cluster local default sleep. If it comes from that it is a get operation and the job token can be validated. So if the request art claim is from a google account, so it is from from documentation.

A

So that is the chat part and the workload you can actually allow it or block it at the service level, but um but as with the x4 certificate. So if that you can block it, so you have a possibility of of operators or security people to set those those security levels, or you have the opportunity in your application to do something with your job token or use the x forward. Client header, in your application.

A

If, if you want to be a workload aware, so we have like a use case internally that really needs to know what what the workload is and what namespace it is to do certain actions. So you have the possibility that, but the nice thing is, you really have the choice you can use both in a mesh, but you have all available in application.

A

You can but don't have to use it, and if you want to mock it, it's just headers, then you're, just mocking header with job tokens or with sx4 certificates, and that's really something the developers like no keys to to handle. Let's just make unit tests with those things and that's it. So thank.

A