Istio IstioCon 2021, 10 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive into Istio Auth Policies

Description

#IstioCon2021

Presented at IstioCon 2021 by Lawrence Gadban.

One of the primary benefits of using Istio is its comprehensive security model, which enables users to express complex authentication and authorization policies for the services running within their mesh. While these security features are commonly used, they can cause confusion and are frequently misunderstood.

This talk will explore the security mechanisms available in Istio and will dive into how these policies are translated from high-level user-facing configuration to runtime policies in the various Envoy proxies that comprise the Istio data plane.

Specifically, we will look at the following:

Mutual TLS and how to configure peer authentication through PeerAuthentication and DestinationRule resources
Enforcing end-user authentication via JWTs with RequestAuthentication resources
Enforcing authorization rules through AuthorizationPolicy resources
Attendees will leave with a clear picture of how Istio’s various auth policies are implemented in the data plane.

A

uh Hey everyone thanks for attending this session, we'll be taking a deep dive into istio auth policies, and so my name is lawrence godvan, I'm a field engineer at solo.io, working on christian's team actually, um and so at solo.I o. We we have a suite of products related to cloud native, api management, application, networking and so on and as part of our glue mesh product, we actually offer enterprise istio support, and so that means that we've been working with organizations at various stages of their istio journey.

A

So there are folks that, are you know just exploring istio, just uh you know, poc and so on. We're also working with folks that have been running seo in production. um Kind of one of the common themes uh with most of them is that in some form or fashion they are.

A

They are interested in using istio security, and it makes sense because it's such a powerful feature with so many different pieces that it's a it's a very desirable featured concept to use, um but even still, there's still a bit of uncertainty in what actually is happening behind the scenes when you're using istio security. So the goal today is to kind of take a deep dive into.

A

You know the various policies you can figure, um but even more importantly, we're going to look at how those policies are actually implemented down in the envoy proxy layer, and so you know that will help just have a you know clear picture of what's actually happening and then, if things go wrong, you'll have an idea of where to look to troubleshoot.

A

So let's go ahead and get started uh to start off with we're gonna kind of take a look at you know the overall picture. So this is the security architecture diagram taken directly from the istio documentation. uh You know there's quite a bit going on here, um so you know as a from the control plane, side, users or operators can create things like authentication policies, authorization policies and those policies will get translated into envoy, config and streamed out to the various proxies that make up the mesh on the data plane side.

A

You know you can have east west traffic from service a to service b. That will actually happen. The actual communication happens through the sidecar proxies themselves. uh Those sidecar proxies can communicate over mutual tls automatically when you have traffic that that actually enters the mesh from say some sort of client or consumer that can that that goes through your ingress gateway and if it's leaving that can go through your ego's gateway and at various you know all of these places you can do things like jot enforcement, tls, mutual tls and so on.

A

So as far as the mutual tls inside of the mesh, um you know, there's been a bunch of great discussion about this already. Just to recap, you know, alongside with the proxies that run in the workloads, there's also an istio agent that provisions the identity for those workloads.

A

So you know, as the workload spins up you're, going to get a identity represented by a certificate and so on, then that happens by the agent communicating with the certificate of certificate authority component of his cod, and so you automatically will have a an identity associated with those workloads, and so let's go ahead and kind of spend a little bit of time. Talking about the mutual tls portion, so by default, istio mutual tls runs in permissive mode, and so that means that workloads will accept both plain text and mutual tls traffic.

A

So we look at this layout here we have the foo name space and in that we have hgtv bin the application and the sleep pod. These both have side cars injected into them. In other words, they're you know participating in the mesh, they have side, cars, etc.

A

We also have a legacy namespace that has a sleep pod with no sidecar, so there's no sidecar injected. In other words, it can't make mutual tls. It can't participate in usual tls with these workloads over here.

A

So in this case, if uh we want to make a request from the sleep application over here to http bin in fu, that would happen over plain text and since we're in permissive mode that is allowed, but also in permissive mode. If you do have um you know, two workloads that are participating in the mesh they do have sidecar proxies even in permissive mode, this traffic will occur will occur over mutual tls. So we have the sleep pod in foo. That makes you making a request to http bin.

A

That communication will happen over mtls, even though we are in permissive mode, and so let's kind of zoom in on this communication here. So this is kind of what it looks like. So we have the sleep pod is making a request to the http bin service, uh port 8000, uh slash headers, and so this hd http bin service is kind of defined over here. This is a you know. The yaml for it um and what's important, is that you know we're the service.

A

Port is 8 000, but the target port is 80., so in other words, the actual workload itself is expecting traffic at port 80.. So you can see that here, and so this request from sleep is a plain text request: it's just http um and there's the sidecar that lives alongside it. That will actually essentially hijack that traffic and it's it's expecting a connection at port 8000, and this is plain text.

A

So then the request or the connection will be established with the sidecar proxy of http bin, um we'll get into the details of this 15006 port here in a second, but this communication will happen over mutual tls and then eventually, what will happen is the connection will make its way to the actual http http application at port 80, as we were discussing with the target port of port 80., and so you know the the actual communication between the applications and the sidecars is plain text, so it's transparent to the applications. It's you know really powerful.

A

You don't have to make changes in order to get mutual tls, so I I you know I wanted to kind of zoom in on each one of these legs, but in the sake of time, we're just going to focus on this one right here: the inbound, the inbound connection of the server side, sidecar. Essentially so we'll go ahead and zoom in onto this envoy.

A

Essentially, so we have an http request being made to http bin, and we this is what that request looks like um we looked at the service definition earlier associated with that services and endpoints definition.

A

You know kubernetes resource, and so this this is what defines what actually, what endpoints are actually serving up that service or you know, backing the service, and so in this case we have a single pod that is at this ip address, 10, 100, 1.9 and the port is 80, because our destination port is 80., so the connection is actually being established to this ip and this port right and it's a istio mutual tls connection, basically, and so what? What now we'll look at? How envoy actually handles this?

A

So in order for so there's kind of two things going on here, one is that envoy handles incoming connections with its listener component right and so on. The left here we have a listener config and that's what this green box is representing a listener at port 15006.

A

But our initial, you know our actual connection is trying to go to port 8.. So this is where the iptables kind of magic kicks in, and essentially it's hijacking that incoming request that incoming connection and redirecting it essentially to this virtual inbound listener. That's at port 15006, and so again there was a talk earlier that you know talked about this a little bit more detail, but basically um envoy will be able to understand what the actual original destination was. In this case it was port 80..

A

So we talked about the listener and you know the listeners. What handles that incoming connection? So then we actually need to do something with it and an envoy that's handled by filters right. So there are network filters. There are http filters later on when we're talking about http, um but so, but we need to actually have our filters to actually do something with connection, and so, if you only had a single filter chain associated with your listener, that would be easy right because you have a listener.

A

You're listening on support as the connection gets established, then you hand it off to the filter chain and your processing happens. But in this case we actually have more than one and that's pretty common in istio.

A

Just because of the nature of how many servers you know, depending on how many services you have in the mesh et cetera, you know that you're, usually gonna have more than one filter chain uh that you need to consider for an inbound connection, and so, when you have more than one filter chain, there is the concept of a filter chain match, and so this is essentially saying. Okay for this given listener.

A

So in this case for the the inbound listener, we need to decide which filter chain to use, and so we need to evaluate some of the properties of the connection and determine which filter chain to actually use. So in this case, we have a filter chain match here in our listener. Config and again, this config is sent by istiod.

A

Essentially, the control plane is sending this config to the sidecars based off the state of the mesh, and so we have a filter chain match here that is looking for a destination port of 80, which in our case makes sense, because that's what we're trying to you know that's what we're trying to connect to uh the transport protocol is tls, because this is a tls, even though it's mutual tls. This is a tls connection um and then this set of application protocols.

A

This is necessary because uh this is essentially how, even though, in permissive mode, you know the the two side. Cars can kind of determine that they're, both speaking the same language as far as istio mutual tls. So because it's the steel, mutual tls, these application protocols are a match, and so we have a filter chain match based off of this connection.

A

That means that we're going to select the filters associated with this chain um and that's how we're going to do our processing, but before we even get into the filters there's there's another component essentially associated with this match, which is the transport socket, and so this transport socket is, you know, responsible for actually doing all of the tls. uh You know ceremonies or whatever you want to call it right. So this is how we actually can do uh tls at the transport layer, and so this this is essentially zoomed in right.

A

So this transport socket has some config. This is kind of the zoomed in view. um This is a little bit hard to read, but it's worth calling out that so with that filter chain match that we had. We selected this transport socket and then there's some config here. That does things like, for example, this required client certificate's. True, you know we're basically saying we're going to do mutual tls, so you know we're not just doing a single one-way tls. You know, as that connection's established the server side is going to say, hey.

A

I need to see your certificate to the client and so we're enforcing that in the transport socket we're also going to validate that cert with the root ca, that's defined for the mesh, we're ensuring that it's a valid, spiffy, cert and so on, and so the fact that this transport socket is associated with this filter chain match means that now we get to use this transport socket for this connection.

A

So then we'll zoom back out and the the next step is really the filters right, and so the in this case we're looking at network filter the http connection, manager or hcm, and that that's really where all of the http processing in envoy happens. And so we have the hdm on this filter chain. And if we zoom in a little bit on the hcm, we have a set of http filters associated with it. And so in this case we have something like istio, authentication, um istio stats and the router.

A

So the router is what actually you know takes the the request, the http request and routes it to whatever destination the the appropriate destination. Essentially, and so, if we're talking about an inbound request, this really is going to route to a you know the application workload that's running inside of the same pod right and that's the last one in the chain, because that's what actually does the routing?

A

So that's kind of the the theme or that's kind of the pattern that we'll be looking at. So you know the listener. The inbound listener gets defined by the control plane and that's sent out to all the con all of the proxies and then based off the configuration that we have. uh You know based on the configuration that we, as a user, an operator have made to istio these chains and these filter chains and so on, are going to be modified to satisfy or enforce whatever kind of policies we define.

A

So, let's take a look at the first kind of policy we'll look at, which is the peer authentication policy and the peer authentication resource is really about how mutual tls is enforced. So you know so basically by default, we're in permissive mode right, but you can change that by using a pure authentication resource, and so you can scope this this policy to various levels. So you can do things like a mesh wide config. You can do a namespace wide config.

A

You can target individual workloads and with pure authentication you can even go down to the port level, so you can have port level config for mutual tls and when you have more than one policy that matches the most narrow scope takes place. So if you have like a mesh wide config and then you also have a specific workload config that workload config will take place, we'll take precedence.

A

So as far as the modes go, um you know we talked about permissive and by default right, so you can accept mutual tls and plain text, and so this this explanation is taken directly from the docs um and we you know in here is called out: that's useful for migrations and really, unless you're completely starting from scratch. You know more often than not: you're not going to be able to have every single workload get a sidecar, and so you basically you're going to have some legacy.

A

Clients left around and so that's kind of the value of permissive mode. Where you can, um you know, iteratively start migrating clients to have a sidecar to participate in mutual tls, and so until you get every workload there, permissive mode allows that to happen. So strict mode is where you're only accepting mutual tls and that's obviou.

A

You know that obviously, is the most secure, because that means that all traffic to this workload is over mutual tls, and so you have a well-known identity you can only if only this is only allowed if you have a valid certificate according to the mesh, and so this is kind of what you really want to get to. um If you're talking about mutual tls, um but it you know, it can be a journey to get there.

A

So that's where permissive mode comes into play, um it's worth calling out that you can also have this disable mode, which kind of shuts off mutual tls, and so it's called out. You know you shouldn't use this mode unless you have your own security. uh One example of this is if you have applications that have their own certificate, built into the image, for example, and it's kind of like a legacy solution, but you still want to run it in the mesh.

A

You can at least disable mutual tls for that specific workload and then, finally, for complete list there for completeness there is this unset, which essentially means inherit it uh inherit the mode from the parent if it has one otherwise, the street is permissive. I actually haven't seen this too much out in the wild, I'm sure there's a valid use case for it, but I haven't. I haven't seen it yet um okay, so we talked about scoping. So when you apply a peer authentication, you can apply it as a mesh wide policy right.

A

And so, if you look at this example, what we're doing is we're creating a pure authentication resource called default, we're creating in the istio system namespace. This is kind of the root name space by default and then we're setting the mode to strict and we're not.

A

We don't have any selectors here so essentially this is a mesh white config due to the fact that it has no selector and that it's in the istio system name space, and so when we apply this config, this pure authentication to the mesh, all workloads in the mesh are going to get essentially they're going to get the effects of the strict mode and we'll look at what that looks like here in a second and then you know, as we mentioned, you can do namespace uh wide config.

A

So in this case uh no selector, but the namespace is foo. So in this case we're applying strict mode to all of the workloads in the food name space.

A

Then you can uh scope it down to target a specific workload itself. So in this case we do have a selector in place and we're selecting the uh essentially we're selecting the http bin application based off label selection, and so that means that now, what we're doing is we're applying strict mode only to the http bin workload in the foo namespace, and then the kind of the most granular is the port level uh scope. So what we're doing here is we're saying to the to the http bin in foo.

A

We want to disable mutual tls on port 80, and so this you know we kind of talked about it earlier. This is really. This is mainly useful if you have some sort of like legacy or pre-existing security solution that uses certificates and you don't want istio mutual tls to interfere with that.

A

It's also worth calling out that, if you're going to use something like this, there is a so you you'll likely need to use what's called a destination rule, so that's kind of a more general.

A

um You know istio configuration that handles client-side config, because most of the policies that we'll look at today apply to the workload you know you're targeting a workload, and that's really the you know the the server side, if you will right so if you want to actually you know if you want to actually interact with something you want to change, how the client's behaving with it, you can use what's called a destination rule.

A

So for something like this, if you were disabling port 80 uh mutual tls, you may want to say: okay, I'm going to create a destination rule that will, when I target port 80 on this workload, I want to use some existing set of certs that are not related to istio mutual tls, for example.

A

So let's look at what actually happens when you apply strict mode. So first, let's look at permissive mode and you know based on what we were talking about earlier. We have a set of filters and and so on, that are selected based off of our filter chain match, and so we can. We have an mtls match and a plain text match.

A

So when istio mtls traffic gets to our inbound listener, we have a filter chain match, that's looking for the port, 80, the transport tls and then the various istio protocols that we were discussing, and so this is a match based off that connection. So we're going to select our transport socket that we talked about and all of the filters associated with it when we have a legacy, traffic or plain text traffic coming in.

A

We have a match for that, because we have the destination port of 80 and the transport protocol rob buffer, and so that rob buffer is essentially saying not tls right, plain text, and so we have a match and we have a filter chain associated with it.

A

So when legacy traffic makes its way to this listener, we'll select the filter chain associated with, with uh this match and notice that there's no transport socket because we're not really defining anything special as far as tls goes so we'll use this one so now when we actually apply strict mode, so let's say we apply the strict mode at the mesh level right, so every workload's going to get this changed. So what ends up happening is now the plain text match the plain text: filter chain match and the filter chain associated with plain text.

A

They will they get removed entirely from from the inbound listener, and so that means that now we're left with just the tls match and you'll notice that the application protocols that were defined here on the tls match have also gone away, because now we don't have to worry about kind of negotiating that we're talking to mutual tls, because in strict mode everything is mutual tls or istio, neutral tls.

A

So now that match is more, is more straightforward: it's if the destination port is 80 and if the transport protocol is tls, and so we're left with that that single filter chain. Basically, so what actually happens when a legacy, client or plain text client tries to communicate with a workload that has strict mode enabled, so the request will come in and you know it's plain text, so the existing filter chain match that we were looking at.

A

uh You know we have destination port of 80 which will match, but the protocol or the the transport protocol is not tls, so we don't have a match here and from kind of the client perspective. You know they're issuing a curl request and then they see this command terminated.

A

You know you're not seeing like an actual, you know, you don't see an http or response code or anything like that. You just see it fail, and so, if you look at these are the debug logs here of the actual envoy proxy and what's happening is you know, you'll see that the connection was accepted, but then the connection handler saying I'm closing the connection, because I there's no matching filter chain found and so essentially by turning on strict mode.

A

We as the sidecar proxy will not even accept the connection, so we're kind of just you know shutting down even any kind of incoming connections if they're not uh istio mutual tls.

A

So and then now, when we have a mutual tls request, connection come in, we have our match because its destination port is 80 and the transfer protocol is tls, and so we use the the chain, the filter chain and the transport socket associated with that match, which includes our you know, transport, socket. That knows about the the mutual tls. um You know we we're going to validate with our root ca, et cetera, and then we have the filter chain, that's associated with it, so we're good to go there.

A

So now we'll look at there's another change, that's made when we're in strict mode. So this is the authentication filter is an http filter. That's in the ch that was in the chain. It was in both of them that we were looking at earlier, and so when we were in permissive mode this uh this is kind of a sample of what the config for that filter looked like on the you know at the at down at the http filter level.

A

So in that case we we had mtls mode permissive and then, when we applied our strict mode now we have mtls with no permissive mode, so essentially we're running in strict mode, and so, when we're running in this mode, um this istio authentication filter.

A

It will validate that the connection was using mtls and that the peer certificate was a valid spiffy cert and then what it will do is it will save off this pure authentication data for use in later filters, and we'll look at this in more detail here in a second just for completeness I'll call out that it does not validate the trust domain. So you can see that this is kind of part of the config right so skip the validate of the trust domain is set to true.

A

This is mainly used when you're using multi-cluster kind of outside of the scope of what we're talking about today, but I will call out that if you do want to do some sort of validation of the trust domain, one of the options you have is by using authorization policies directly to do that.

A

Okay, so again, we're looking at some debug logs. This is uh with uh this is the sidecar proxy for http bin. Remember the request was coming from the sleep pod, um and this is the istio authentication filters debug logs, so you can even see here right, so the mode is strict, ssl, true, etc.

A

What we're doing here, we said, you'll see this set here from x 509 as this cluster.local namespace foo uh sa service account sleep. This is essentially the the spiffy or the workload identity of the source, and so we're setting the principle from that from that, and then you'll also see that we're calling out we're saving some dynamic metadata and a note you'll see that in that dynamic metadata we're setting the key of source.namespace to be the value of foo, which is the namespace of the request of the initiator and then the principle, the source.

A

That principle is set to that identity that we were talking about. And so what ends up happening is the issue. Authentication filter uses another envoy component called dynamic metadata, which allows filters to essentially communicate with each other.

A

It's going to save that data for use later in, for example, authorization policies and so we're setting a source.principle so in our dynamic metadata for iste authentication, we're setting source.principle to be that identity and we're setting source.namespace to beefu and that's how we're going to be able to access it later on.

A

So that's kind of it for the pure authentication piece. Let's take a look at request, authentication, and so now we have a request, authentication resource for that.

A

So this is really all about authenticating, the actual individual requests and right now, the main way that you do, that is with a json web token or a jot, and so what you're specifying here is kind of the where, where the token comes from so by default, that's going to be the authorization header with the bearer prefix um the issuer of that jot and how to actually get the json web key, set or jocks uh of that associated with that with that uh provider or that issuer, and so this is basically where the public key is so that we can do verification on the jot and make sure that it was signed uh by the correct.

A

um You know private key basically and so istio will will take this and translate it to some envoy config and then we'll. Actually, um you know verify various comp various pieces of of the jot itself um again similar to how the peer authentication work. You can scope this to different. uh You can have different scopes for this, so you can do mesh wide namespace wide or workload wide, and you can have multiple of them. Multiple of these, the the end result of that is that they're, aggregated and you're.

A

What you're really doing when you do that is you're, saying I have more than one provider. So if I present a jot for, if you have say two policies, one from foo and one from bar or whatever, if I present a jot, that's valid according to either one of them, the request is allowed basically and so again, here's a view of kind of the the scoping right. So in this case uh again in the istio system, namespace with no selector. That means it's mesh wide.

A

So this request, authentication config, gets sent to all of the workloads all the proxies in the mesh.

A

If we do a namespace wide, so no selector, but a non-root namespace, so name says foo all of the foo workloads will get it and then, of course, you can target individual workloads, so in this case we're targeting um the http bin app in the food namespace.

A

So it's worth calling out that you can do request, authentication at the ingress layer right and normally you know it's a pretty common thing to want to do this, because request authentication usually corresponds to a client like a human user or a client, um that's living outside of your outside of your mesh, and so it makes sense to do those at the edge right, and so you can apply a request: authentication config at the ingress gateway, for example right.

A

So here's an example where we're applying a request, authentication in the seo system, namespace, that's targeting the ingress gateway and then we're defining our our issuer and where to get that um our jocks data, our public key, and so in this case now. What we're defined is that the ingress gateway is what's going to actually perform. This request authentication.

A

So you know if we look at the kind of the data path, uh the data plane, the request path, uh users, making requests with the with a jot to the ingress gateway, and now the authentications can be formed here at the edge. So if it doesn't have a valid request, it doesn't have a valid jot. That request won't be authenticated even at the edge, and so it will be denied at the edge.

A

But if it's, if it's allowed, then it can be forwarded to the upstream services within the mesh. It's worth calling out that, if you do this, if you want to perform additional job-based uh logic right either, whether that's you know, writing authorization policies based on the jot or if your applications themselves rely on the jot. So you have actual you know. Security features built into your application.

A

You want to forward on that token, so, as far as how that actually is enforced, we're adding another filter to the filter chain, we're adding the android jot filter, and so this jot authentication filter is what performs the actual enforcement. So this will be added into the chain with associated listeners, um and so in this case you know this is kind of the translated config.

A

For it of note, you'll see this allow missing requirement, so the requirement on the jot filter is a provider of what we've defined or allow missing, and so this means that, if you don't present a jot at all, the jot filter will not deny it will allow it. So that's something to to think about so, for example, if you have a invalid jot whether it's because the mission the issuer doesn't match, the signature is not valid, it's expired, uh it doesn't have the correct audience.

A

If that was configured, it will be denied, but if it is valid or if it doesn't has a job, it doesn't have a job at all, it will be allowed, and so um then, when we're doing request authentication, we also modify the istio authentication filter to now perform um origin authentication and so there's actually not a whole lot of enforcement going on in this authentication filter when we're doing origin or request authentication, but we are doing um something useful as far as extracting the request principle.

A

So, let's take another look at the logs of the sidecar or the proxy. That's handling the request, authentication so we're processing this jot. Payload you'll see that we're setting the principle based off of the origin, so there's the john.

A

So we have flu global corp, dot, io, slash john- and this is the this is the origin principle that we will then save for later use. So this is the issuer and the subject, and so this is saved again in dynamic metadata, like we talked about before so um now that the iste authentication filter can handle that now, this authenticator gets access to that data.

A

It's going to store that in in the dynamic metadata for use later on and then of note where we're kind of we're kind of storing the full claims of the jot as well. So that's how you can do really. um You know complex or really fine-tuned uh rules based off of the jot that you are validating or authenticating, and so quickly, let's run into the authorization policies themselves.

A

So the there's another resource that allows you to control. um This is actually access, control, right and so, in this case, you're defining a set of rules. That say, say, for example, from or to so based on the source of a request and the operations of the request, and, additionally, some optional conditions with the win field. You can do not allow or deny um access. So what that means is that now you can um create, allow or deny rules based off of any of the any of the values that we were just talking about.

A

So basically, what was the source of the request based on the peer authentication or the source of the request based off the request authentication, so authorization is kind of done in an implicit enablement fashion. This is what is called out in the docs, so when there's nothing present everything's allowed, but once you start adding rules, it will change the behavior.

A

So if you have one or more allow request now you um or one or one or one or more, allow a policy that means that now everything will be denied unless it was explicitly allowed um deny policies are evaluated first, meaning if you have a deny rule.

A

If you say you have a rule that matches and it's for an allow, if there's a similar rule that matches for deny in front of it, uh regardless of the allow rule, it will be denied and multiple policies will be aggregated, meaning that you, you basically don't have to define all of them in a single policy.

A

You can have more than one of them so with kind of the semantics, it's kind of a best practice kind of recommended to use something like a deny all rule where in this case this is a mesh wide config and this empty spec means we're going to deny everything, and so that means that you know rather than kind of the implicit enablement functionality. You're very uh you know very um being very heavy-handed and saying that nothing is allowed at all ever unless I authorize it.

A

Maybe that's a little too heavy-handed for your organization, but it's something to think about it. Kind of it kind of makes the mental model a little bit easier so quickly, let's kind of check in on time, yeah we're running kind of slow low on time, so just really quickly. How this actually is implemented is another filter in the chain, the r back filter from envoy, and so this is how you actually enforce the policies. This is the actual enforcement point, um and so this is the our back config.

A

So we're looking at the header and get so so in this case we're the permission is a match on the method. Pseudo header, if it's again- and so our curl- is a basic get. That's that's correct we're good to go there.

A

Then the principle is going to look in our dynamic metadata in the istio authentication filter check the request, auth principle key and then check the value of it to make sure that it's the correct uh principle that we we defined, and so since in this case it is, you know, that's in the in the metadata.

A

Now we have um that's successful and so the request will be allowed, and so it's kind of a quick journey, um just a quick plug for if you're interested in some of the stuff we're doing at solo come check out solocon next month. You can sign up here at this address here, but with that we'll go ahead and stop and we'll maybe do some q a if we have time but yeah. Thank.

A