Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Zero Trust Service Mesh with Calico, SPIRE, and Envoy - Shaun Crampton & Evan Gilman

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Zero Trust Service Mesh with Calico, SPIRE, and Envoy - Shaun Crampton, Tigera & Evan Gilman, Scytale

The promise of a service mesh is to be able to delegate the hard networking problems to a uniform set of proxies and controllers. An extremely important networking problem is securing traffic within the mesh. Service meshes based on the Envoy proxy are very popular and there is a large diversity of implementations, including many home-grown solutions that focus on routing but may not include security. Shaun and Evan will demonstrate how to enhance your service mesh to follow the Zero Trust network security model using SPIRE and Calico. A Zero Trust Network emphasizes resilience to compromised services, hosts, and the network itself by treating every network connection as potentially hostile. Secure connections are established only when backed by strong cryptographic identity and approved by fine grained policies---provided by SPIRE and Calico respectively.

https://sched.co/MPe3

A

Hello. Everyone please join me in welcoming onstage, German, Gillman, engineer, Saito and Jean Cramton senior software engineers Tiger. They will tell us about service mash zero to our service mesh with calico spire and then boy.

B

Hi everyone, yes, so I'm, showing this Evan hello. They said we're gonna talk about zero trust, service, mesh or kind of building a zero trust service mesh with Kelly codes by an M boy. Now, as you can imagine, with a title like that, we've got quite a bit to unpack so Evans gonna start by giving us a rundown of what zero trust is. Then I'm gonna talk a bit about service measures in general, then we're gonna dive into spiffy and spire calico and pull it all together for a demo. At the end, Evan.

C

Asome Thank You Sean see ya, so mr. hero trust, this has become pretty pretty unfortunate buzzword over the last couple years and, and so many of you may have heard of it in the past, but despite it kind of being fairly well known at this point, a lot of people still aren't quite super familiar with it with the concept behind it and the simplest way to describe it. I think is just that. It's a simple statement that hopes that data and entities shouldn't be trusted by default.

C

So this idea can be applied to like many aspects of computing, but if you think about it, a little bit deeper can't really be taken to the logical extreme, because in the end you have to trust something right. It's very difficult to get a computer to do anything useful for you. If you don't trust anything about it, you think, like the execution environment or the hardware, it's running on I think it's like this, so you know don't get me wrong.

C

There is some work going on in these particular areas, but I think that for our purposes today, it's useful to kind of like pull back a little bit and be a little bit more specific. So today we're just going to be talking about the zero trust networks, in particular, so zero trust network, just kind of takes this GeoTrust concept and applies it towards network design. So what does that really mean? It just means that all those entities and data on the network are strongly authenticated and authorized.

C

So the data has encryption, applied and integrity, protection and the requesting and serving entities are proven to be legitimate and they're also proven to be allowed to actually request or serve said data and there's a big difference between those two things as you'll see later on.

C

In this talk, um so this is a pretty far departure from you know what I would call traditional network design where networks have this kind of crunchy shell and this soft gooey center, and then you provide just like rigorous protection around the perimeter, and you form these like trusted zones in the middle, and then you wire those trusted zones together. So this is kind of an example of that traditional network design, where you have two different data centers with a site-to-site tunnel in between and then each one has this kind of perimeter protection.

C

You know, which you know keeps the bad people outside and the happy ones inside. The problem is that this inevitably leads to like really poorly secured hosts, because the idea is that the network on the inside is safe and, of course, this practically all networks have some sort of entry point all right like it can be.

C

A public web server that was accidentally exposed can be purposely exposed if it was a product or something like that it can be a VPN server, a jump host, even air gapped networks tend to have entry points right and, and those are sometimes the worst offenders, because the belief and perimeter security on those kinds of networks are so strong, but whatever the entry point is, it can definitely be abused. I assure you so most networks, don't really look like this.

C

They kind of look more like this right, so the key thing to understand here is that there's always some way to get in right and when a host is compromised on that network, a lot less than if there's internal controls tend to be really really poor and the end result of that is that it makes lateral movement throughout the network super trivial. To do so, a single compromised host becomes multiple compromised hosts and all of a sudden, this attack is much worse than it should really have been in the first place.

C

This kind of thing happens all the time. By the way. It's not, they made up story right. This is a key element and the target attack that everyone heard about a few years back where the attacker is famous. They came in through the HVAC system right.

C

So this is all a result of like this poor internal security posture, because it's assumed that the network and the hosts that are on that network or are safe and trusted, and when this happens, we all loose right and not only do you lose, but your customers probably lose out too. Luckily, there's a simple answer to address all this stuff and it's just don't trust the network to do the right thing and don't trust a host just because it's on your super, safe private network.

C

So, rather than building your networks like this, where they have this crunchy outside in those gooey inside, you have to kind of imagine that there is no perimeter right and that there are bad people all around, including inside your network. Perhaps already so you build these networks to be crunchy on the inside right and when you take this this, you know zero trust, idea or model, and you apply it towards network design.

C

This is this kind of posture is pretty much required in order to meet it, so we don't assume that there are no entry points or that the perimeter is is infallible. Instead, we acknowledge that someone's going to get in and we build like tight controls around each service. So when someone does get in damage is much much more limited than it would have been in the previous posture. So this like sounds great in theory, but there's one big problem with it and it's how do I manage all this crap right?

C

All of a sudden, you have to have authentication everywhere credentials. The applications have to speak TLS or some sort of other. You know wire protection protocol and you have to have access control lists and those things have to all be managed like who should be talking to what and those control lists change all the time right. New services are coming and going new dependencies are being introduced.

C

So you know you have all these knobs and have to be a constantly twiddled and maintained in order to keep this state up right, and we certainly don't want developers to have to deal with this thing. You know it's wrong on dell. L developers means like well, you have to train them to do that. This is a whole other skill set, and it's quite like a deep area and security in general right. Some of these configurations can be really really easy to screw up.

C

If you ever configure like a TLS stack or something it's definitely non-trivial, we want these developers to focus on business problems. We don't want them to have to worry about, like you know, authentication and access control. This and all this other crap right. It's not to mention that you know not. All apps can actually be modified to do these things. Very. A lot of them are packaged up, or maybe the person who wrote it is no longer there or whatever.

C

So this is where, like the concept of a service mesh comes in, and the service mesh includes a transparent proxy that is like adjacent to every workload, and this proxy then becomes like a really good place to implement these security controls. So it allows you to like slip those controls in underneath the application without actually having to modify it. So you get all these like nice and fancy things that we want in order to build a zero trust network, but we can do it without necessarily like directly modifying the apps. Yes,.

B

So service mesh another one of those buzzwords. It's.

C

Definitely a buzz word, yeah.

B

Let's unpack that a little bit sure.

B

Cool, so just just kind of rolling back a little bit when we talk about a service mesh, let's get back to basics. So let's say we have an app running in a pod over here and it's accessing a service over there. This is how most of a prototype start right. We we set up a simple HTTP request. We get the data, we want job done, except one. We try to move this into production. We start caring about other things. So we start caring about discoverability like how do the applications find each other.

B

We start caring about load balancing. So how do we scale up? You know more advanced things like a bee testing and things like that too. We start caring about observability. So how do you see? What's going on in your network and a service? Mesh kinda comes out of out of these ideas, but it's also a good place to do the things that are really relevant for our talk, so encryption authentication authorization so encryption to keep your data from being Easter opt on authentication so that you know who you're talking to and authorization.

B

So you you can lock it down and say who can talk to who, as Evan said, a cervical mesh works by putting a proxy really close to each of your workloads. So the idea is that your application continues to make its simple requests: the proxy intercepts it and then, between the proxies. We try to do best-of-breed versions of all of these things, so best-of-breed encryption with mutually authenticated TLS and things like that as euler guessed from the name of our talk. mmm Boy is the proxy that we're using here.

B

So envoi is a fairly light proxy. It runs inside your pod as a sidecar intercepts the request really close to your application before it leaves the pod, and that means that even the host namespace on on the host doesn't see the unencrypted traffic. An envoy basically does all of the above. So it does encryption authentication, it can enforce policy decisions, but whenever you're doing something like encryption and authentication, you need a whole bunch of certificates. Right, Evan,.

C

Yes, definitely not just certificates, you need an identity alright, and this is kind of where spire comes into the picture. I'm spiffy too, of course, so spiffy inspire are two different projects, but they're very much related spiffy is the logo up on the left. Here, spiffy defines workload identity, so that is like what is a workload identity right? How is it expressed?

C

How do I prove it right? How do I get one? How do I Fedder a tit so on and so forth? At the end of the day, it's really just a bunch of markdown documents, spire.

C

On the other hand, as by process of elimination, the one on the right is an implementation software implementation of those pipi specs, so spire is just one of these implementations are several, but that's the one that we'll focus on today, so spires job is to handle the secure issuance of these spiffy IDs right nurse nifty identities rather, and these identities have a finite lifetime, so by necessity, aspire also automatically handles the rotation of these identities and, after a lot of thought, I think the best way to describe spiffy is kind of like a passport right.

C

Everyone knows how to read a passport they're all generally the same. They all have the same size. They are the same funny-looking barcode at the bottom. In the same place, they have a picture et cetera. The dimensions of a passport are very well-defined they're hard to fake all right, and they just prove who you are? They don't necessarily say that you're allowed to enter a country.

C

They just have your name and what country you're from, and it does a pretty good job at that, no matter where you go so in that vein, spire is kind of like the passport agency right at first, you don't have a passport, you need to get one, and then it expires after sometime. You need to renew it all right so aspire is the implementation that handles the issuance and renewal of these spiffy identities or passports.

C

If you will aspire comprises two primary components: the spire server and the spire agent, the spire server acts as the authority, in this case the certificate authority or CA, and that's responsible for controlling the issuance of these identities. It also houses the authoritative key, which is used to sign them and prove the validity of them. The agent, on the other hand, manages this certificate lifecycle and also figures out which certificate to give to which envoy.

C

So you can see from this diagram here that spire lives inside the control plane, so it's really only needed when envoy boots and when it has to rotate these identities, so um I can still run even if there's a spire outage or something like this, at least until those identities expire. That's the agents job to track those explorations and as they're coming up it generates new ones and pushes those new certain tomboy, but because this renewal is automated, the lifetime can be fairly short.

C

So you know again, this is a trade off it's a it affects, availability right, but it's up to you right. Some people choose one day two days a week, something like that. Some people choose 12 hours, but since this is fully automated like this, we don't actually need to store any of these keys, because we can always just get a new one if we need to which means that you don't actually need any long-lived credentials in the data plane, which is really nice.

C

In fact, spire knows how to rotate all of the keys. Not just these. You know, and and identities I can even rotate the route keys too. So that means that, under certain configurations, long live keys aren't really required in the control plane either. No I'll stress that spire is SuperDuper flexible, so you know how you configure it and deploy it. There's a series of trade-offs right, but in the end a spire could provide this really highly automated spiffy base PKI system that delivers and rotates these search for your fleet of envoys.

C

So here now you have all these envoys they've been all issued these nifty identities, and you can use these nifty identities to do mutually authenticated TLS between the envoys. But really this is just the beginning. Right spire knows who is who, but it doesn't necessarily know who should be accessing what yeah.

B

That's a different problem: yeah.

C

It's a very different problem: it's the difference between authentication and authorization. So Shawn will you tell us a little bit about the authorization part sure.

B

So, just going back to our simple example: we've got M go in there, we've got the spiffy identities provided by spire, and we've got mutual TLS going between the two. So you know it's all great everybody knows who C, but what about this request coming in from another pod? So this is another pod in our system. It has a valid identity, but it has no business talking to this particular server. So maybe this is a compromised pod.

B

Maybe this is a pod that is running in staging and someone programmed in the prod database credential, something and it's accidentally trying to access the the the production database either way. We want to lock it down minimize our attack surface and just only allow the requests that we want. So that's what a calico comes in so one component of calico is is Felix, which is our host agent and Felix's job is to calculate the precise policy that applies to each pod.

B

It does that by reading the Calico policy from our data store and the Calico policy that we care about here is what we call application their policy so using our application. Their policy, you can specify things like only allow HTTP GET, so you effectively make something read-only and you can specify details of the service accounts that you want to allow requests from.

B

So you can say, I only want to allow requests from, like my client service account and then that service account will be backed by a spiffy identity, and you can lock it down to the individual HTTP paths if, if you're, using HTTP and as as envoy, rolls out additional additional features for understanding different protocols, we'll be able to add additional additional match criteria in there. So Felix takes that policy. Distills it down for each individual pod calculates the optimized policy and then it hands it over to our other component, which is called de cast ease.

B

Now the cast ease is our android plug-in and the reason is wearing a Judge Dredd mask like that is because the cast ease is the ancient Greek judge or decider.

B

So the cassadee's takes a cache of the individual policy for a particular pod and then, when a request comes into envoi and boys going to authenticate the request, figure out the identity of the of the requesting service, and then it's going to pass the details of the request, along with the details of the identity over to DES Cassidy's de Cassidy's, execute the policy that Felix has calculated and returns a response like yay or nay.

B

The reason we have a split between Felix and DES Cassidy's is because the Cassidy's is in the data plane. So de Cassidy's needs to be highly available. Felix is our country, one of our control, plane components so Felix can restart and that won't affect your network. It's similar to spire like it's Felix, is only needed.

B

You make changes to the to the data model and Felix monitors the policy continuously and and updates it. So Felix is how you kind of handle all those 10,000, a CL knobs that everyone was talking about, bringing that all together. We can lock out that request coming in from a random pod and eventually we end up with. Therefore, 2 less of an authorization over spiffy identities issued by spire cool. Thank you of a demo right. We do have a demo, let's check it out cool.

B

So, like all good demos, we're gonna rob a bank, I love robbing banks, and this is the setup. So my colleague Spike's gonna open his web browser there and I want to give him a big shout out because he was the guy who built this demo for me. So spike opens his browser and he's going to hit the yet another online bank customer portal. That's gonna, go via a second micro service which is in charge of getting the particular summary details for for Spike's account.

B

That's going to hit the database and we get a request coming back, showing the showing his balance.

B

It wouldn't be a service mash demo if we weren't running a service mash, so we've got envoi in running in all the pods and spires going to be injecting those lovely spiffy identities and, of course, we're gonna run Felix and the Cassidy's so we're ready ready to roll with policy. Before we add some policy, though, we're going to use the good old fashioned keeper tool, exec zero-day, to get into this, this customer facing pod so we'll get root in the customer facing pod.

B

Then we're gonna directly access the database and see if we can give yet another online bank a bad day, then we're gonna load up the policy and lock ourselves out again cool sound.

C

Good yeah: let's do it cool demo time.

B

Afraid I have to go into settings and switch to mirror mode.

C

Stupid presenter tricks, yeah.

B

Okay, there we go so like, like all good kubernetes demos, we have a whole load of yeah, Milles and I'm just going to start applying those so I'm going to apply the Calico llamo. This is just a basic off-the-shelf, Calico llamo, with application. Their policy enabled that's like a tick box option in our Docs, then we're going to apply the spire server. So that's that first component, that Evan was talking about and I'll pull that up. Okay. So he wants to say anything about that. Oh.

C

Yeah sure so this is the spire server deployment yeah mo it's a pretty typical spire server deployment. That I think the most interesting part here is just the plugins, so you can see that we have a pluggable data store and inspire. So this is demos just using sequel, Lite, basic sequel, Lite plugin, but in kind of snap different things in here. If you need to, we have this plug-in called an Oda tester, which is like when new agents or nodes come online. How do they identify themselves to the spire server to begin with?

C

How do they bootstrap that relationship? We call that metadata station and here we're using kubernetes service account tokens, and finally, we have this upstream CA plug-in here, which is allowing spire to actually plug into like an existing PKI system. If you have one in this demo, we have this wired ups like a dummy key that we use to bootstrap the demo just so that it's a little bit cleaner, newer versions of spire. The server actually knows how to inject this bundle as a config map into kubernetes. So this bootstrap key isn't really required.

C

Demo is just a little bit older than that in that regard, thanks.

B

Evan go on to the 14 manifest, so this one just applies the auerbach that spy needs there's nothing fun in there. So I'll, just like that. One I'm gonna play the agent and I'll pull that up again. Yeah.

C

I mean this one looks kind of similar to the server configuration it's just the you know the agent side of things so again, you'll see this node a tester kubernetes plugin, because the agent has to know how to challenge the server. Has the two things gonna have to match this key manager. Memory plug-in is basically saying, don't write any of these keys to disk just keep them in the memory for now and then. Finally, we have this workload a tester plug-in, and this is how spire agent is able to identify which envoy is calling.

C

So in this particular case, we're using this kubernetes workload, a tester where the agent knows how to talk to the cubelet and get a bunch of information about the pod, and things like this.

B

Thanks I'm one thing and it just came to mind: is we used to have a version of this demo that we that we ran with it with another service mesh and in that version we used to demo like stealing a certificate, and then we thought like I. Could we work that into this demo? It's like it's in RAM, like it's really hard to steal, like, even even if you have like privileges to enter the pod, but you need to be like crawling through memory to find the certificate, so we couldn't steal our own certificate right.

B

So that's that's applied next that we're going to apply might create the identities in the in this bio database right just.

C

Teach aspire who is who right? So you know this envoy is you know foo, and this envoy is bar, and here are the spiffy IDs that that you should be assigning it. We've looked at that workload, a tester plug-in on the agent before that is kubernetes based. That means the agent knows how to identify workloads based on kubernetes primitives.

C

So you see these things called selectors here, where we can use these kind of kubernetes ideas to identify particular workloads, because at the end of the day they don't have a secret or any other way to identify themself. This is what we use and you can mix and match. These things doesn't have to be just kubernetes. It can be unix primitives, it can be all kinds of other things, too. Cool.

B

So now think we're ready to apply the bank itself and while that's starting up.

B

So standard kubernetes stuff we've got services for each of different pods. We've got a service account for each pod and then because we're configuring envoi manually, we've got an envelope a config map for each pod as well. So this one's for the database we're telling envoi which, which like TCP ports to listen on things that one actually we're telling it to manage, HTTP and which URLs it should serve, we're telling it that it should do or thent occation by the cast ease and we're telling it to require certificates.

B

So this is where the M TLS comes in, we're telling it who it is so it needs to know its own identity and, if you were doing this like for real, like moving into production, you'd want to be auto-generating all these. All these things right and I think you've got some working yeah.

C

B

C

An admission controller coming up which handle a lot of these things, at least on the spire side of the house, yep.

B

And then we're saying we're putting in SDS config thing: that's the API that spire connects in on and we have like mounts for the DES caste socket and the spire socket, and things like that. Then we have the same for each of the other services, so one I won't bore you with all the detail. Now. If everything went to plan, I should be able to open the browser and we should see exactly what we've got before so great.

B

So if I hit ctrl R, so I am refreshing that just to check yeah, so service is live now and we can see spikes balance.

B

But let's see what happens if I jump into the customer pod, so hopefully they've left bash in there. For me, oh I need a TI in there.

B

There we go so hopefully you know they left bashing their customer-facing, but I was a little bit when they do that right and looks like curl. Is there too? So that's really useful. Let's see if they've got their nice discoverability set up so hmm yeah database I'm getting responses and I guess I'm gonna guess it's gonna be a @ CD database. So let's see.

D

If we can get some keys, ooh.

B

And if I tack recursive equals true all right boom, so we've got rich yeah exactly so, we've got the whole of yet another online banks database dump to screen it's not a good day for yet another online bank right. So, let's see if we can lock that down.

B

So if I apply, the Calico policy and I'll just show you that to show you that policy I played. So this is a calico network policy. It's a global network policy so that that applies across namespaces, which is something you can't easily do and in keeping that ease and the two things I sort of call your attention to one is the the selector here, so we've labeled our pods like customer summary and database. So this selector here is based on those labels and it tells Calico which pods that should apply which policy to so.

B

This is what Felix uses when it's trying to figure out which policies apply locally, and then we've got some ingress policy here. So on the customer.

B

Pod we've said allow ingress and allow get only so where we're not allowing puts or anything there, but perhaps more interesting, like if I go to the the summary and database pods, this row gets. It gets even more relevant for our talks so applies to the summary and we've got an ingress rule there. That says only allow from this specific service account and that specific service account due to the way we've structured.

B

Our spiffy identities is going to map to the the spiffy identity for that one, so it will be authenticating the their particular so well, we'll piggyback on the authentication for the specific service account. Then we'll authorize that that request.

B

We haven't worried about egress policy in this particular demo. Okay, so if that will apply, just fine I should be still be able to access the web services still up. So it's a good good test, and now, if I exact back into the pod and try running what I did before I get in that outlet, but if I put a dash V on there there we are not rich we're, not rich, so yeah, that's the story of how we didn't manage to rob the bank.

B

In the end we tried yeah so that 403 forbidden is coming back from m'boy. So you get you get a fairly clear response. That tells you why your request wasn't allowed and that's what you'll see whenever de Castries denies a request?

B

C

Think that's all we have for the demo right, yeah.

B

C

If you guys want to learn more a couple, links link to the spire project linked to the Calico project, and you can actually download this demo and run it yourself, there's a link to the demo yeah Mille. There feel free to pull it down, run through it, and let us know what you think. Thank you. Everybody thanks.

C

I believe we have time for questions. Is that true question time? Oh I think you need to pass this right.

E

Hello, you mentioned in the demo that or either before the demo, in your presentation that you're using Calico policy controller to secure well using the whole solution to secure last four through two seven. Are you also using the post controller to implement any security at layer 3 so between pods.

B

So in this particular demo, I don't think we had anything in there, but you absolutely can so you can. You can have kind of typical Calico policy like not the application there stuff and that runs a layer three based on IP, so you can have both of those running, and maybe you get you know defense-in-depth from that, like this certain types of attack or someone steals a certificate or something you might be able to lock them down, but it's kind of not really the theme of our talk because we're talking about zero trust. Networking here.

F

E

You think calico 4c and I yeah.

B

E

B

Don't have to be using calico for your CNI plugin for this, so calico Calico has a c and I plug in, and you know we support various types of networking and an overlay, but this is all in our policy engine and the policy engine works with various difference he and I plug in. So we work with like AWS, SC and I plug in and the the EOC and I plug in and very various other ones that we were flying on as well. So.

B

Questions down here.

C

G

Am curious, those aspire support, multiple roots of trust, say the bank has different, see, ace.

C

Yes, it does so depending on how you want to set it up. You can have multiple roots of trust and kind of the same. What we call trust domain or you can have disparate rates of trust, which federate between each other, that's something that is fairly new and is kind of in the works now, but it'll allow you to kind of have like multiple, disparate domains of trust, with multiple disparate roots that have like a loose coupling between them. Oh.

G

Fantastic looking forward to it, thank you. Who's.

A

A

H

Was wondering about our spy era, exactly a position compared to sto Citadel, and is it possible to do the same thing here using Citadel for the MPLS certificate deployment I.

C

Believe it is possible, do Citadel relies on service account. Tokens to authenticate envoys aspire doesn't have that requirement, there's also an integration that we have where you can use a spire server in place of Citadel server and then use this do note agent instead of spire agent, so there's a lot of different possibilities here. The key thing to understand is that Citadel is a very, very purpose-built and works in a small number of cases or aspire is kind of a general purpose solution.

C

We can work across different kinds of environments and different platforms, and things like this.

H

You are using invoice as a data plane, but is it possible to would this work as well with data plane, implementation, yeah.

C

So that the trick is just that, you need to get parody on the API that spire uses to push certs into the into the proxy. So here spire is supporting STS API, which is an envoy specific API, but we also have an engine X module which knows how to speak the spiffy workload API and can talk into spire agent and pull those certs out the same way. So it just depends on the API support on both sides, and even if there's not there, you can kind of build some glue in between those two things. Okay,.

I

How do you establish trust between Felix and calico, like how do you verify the policy that you get? Is the.

B

Question, how do you establish trust between sorrywhat and.

I

Calico Velux well, the agent Felix.

B

B

We have a slightly awkward problem because we're kind of an infrastructure layer, so we kind of can't rely on all of the stuff that that spire provides, but we do support the like normal security primitives for accessing like the kubernetes api and we support TLS for accessing at CD. But it's a bit more of a manual process or you know we have some upcoming work to do a do, an operator for calico which will automate setting up a lot of the certificates and so on.

B

It's a little bit frustrating that we want to work in in networks that have you know they don't have the spire setup yeah, because spire as a service that might run on top of calico to serve to its pods. So we we have to kind of do some of this stuff manually, but we we use, we support certificates and so on, for establishing that trust. All.

I

Right. Thank you.

G

So how would you go on and debug network issues if you write a policy that isn't valid, it can't match on based on selectors, that force it etc.

B

So you can turn up the debug logs in the castes and it will explain the decisions it's making. The caste is also logs, I. Think every decision, so first you'll get the 403 response that will have envoi in there and it will say this is a 403 from envoi and once you know that you've got a 403 from envoi. You can go dive into the de castes and then boy and get more details about what happened. And if you crank up the debug log, you should get get information about exactly what happened.

F

How does the health check look for something like this? So how do you connect between cubelet and the actual port.

B

So I think what happens there is cubelet looks like an outsider, so normally in Calico policy, but not the application less stuff. We just wipe this cubelet and allow it through with application, their policy I'm, not a hundred percent. Sure sorry.

F

D

Any tips on how to safely roll out spire updates or config changes.

D

C

So you know I've mentioned earlier that spires this control plane component. So you you can't like tolerate some downtime there. A few few fat fingers screw something up while rolling an update out. That's something that is okay to do, but in general we recommend that you upgrade the spire server first and upgrade the agents, but we do maintain like compatibility between major versions of spire, so it shouldn't be too difficult to roll out without it hit.

C

But even if you do encounter a problem or something like that, you know if your TTL, if the lifetime of the certificates is one day, you'll have 12 to 18 hours or so to clean things up.

A

Any more questions we'll take a while to go there.

B

It's kind of the same story: the Calico is where we, we have a strict separation between data plan, components and and control plan components. So.

J

What's the overhead on the overall application performance with authorizing every single cross support request so.

B

J

The overhead on the application performance, because, basically you have to update your expected Leighton season. Yes,.

B

Sir, so there is, there is overhead and there's an extra bit of overhead from the small network, hop to to envoi itself and then there's the processing overhead of doing like an TLS or something like that and whatever approach you take. You're gonna have the MTA less overhead and we have some work in progress to optimize the connection to 2n voice so that we can use some of the features in the kernel to to kind of bypass that socket and get the package directly to to envoi. So.

J

B

Does add some overhead I, don't know exactly how much, but but yeah can't get away from that.

A

I think we ran out of time. So thank you.