Red Hat OpenShift Security | OpenShift, 26 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Zero Trust based Authentication with SPIRE and SPIFFE Frederick Kautz (Doc.ai) Bobby Samuel (Anthem)

Description

OpenShift Commons Briefing
Building Zero Trust based Authentication with SPIRE, SPIFFE, NSM & OPA Frederick Kautz (Doc.ai) and Bobby Samuel (Anthem)
hosted by Diane Mueller (Red Hat)
2020-05-26

A

All right, everybody welcome back to yet another openshift commons briefing. Today we're going to talk about building trust based, authentication with a whole bunch of open source projects, spire spiffy, nsm oppa and two openshift commons members, frederick coutts, from bobby samuel from anthem. Ai are gonna, um walk us through some really interesting content here and maybe lay out what all of those wonderful open source projects are that they used to make this happen at anthem and other places. So I'm going to let frederick and bobby introduce themselves. Take it away.

A

Talk for probably 30 40 minutes depends on how much content they have and then give us some time for a conversation at the end. So frederick take it away all right.

B

Frederick, do you mind if I'll I'll start.

C

Sure um all right.

B

You're, first, all right all right thanks um so good morning, um everybody and thanks for taking time to join with us this uh today. I hope we had a lovely memorial day weekend and uh frederick and I are going to tag team here. So what I'll do is, let me set up the business context into sort of the. Why why? Why did we start this? Why are we on this journey and why is it important to all of us, as well as to people that you know, carry insurance and people that work in this space?

B

Why? Why is this important? So um let me set that up and then frederick will walk into some of the technologies that we're working on so the project. If you could hop to that first slide, I will um I'll just set us up, so um I don't know where your eyes go when you look at this slide, but um I was I was here at anthem and I only see just three words on here: 2015 data breach. um That's I I lived through that here at anthem and it was.

B

It was not a great time and so um anthem since that time, um and some of you saw it in the news and it was over 25 million.

B

You can go search on uh on um your your favorite search uh search engine, it's 25 million or so members had their private uh health and uh information taken, and um you know taken out and uh we're still facing the repercussions from that, and so, as a result, anthem's reaction was to beef up traditional security um and to to put those uh traditional security in place at at one point in time.

B

um I believe it was um a little over seven hops to get through our firewall to get to an external uh website, and so we we took a traditional model and we bolstered that or galvanized it even more and as we've looked at the it's, it's not sustainable long term and uh we're taking a second look at this. So health care itself is just the cost of health care. Those of you um actually many of you that are in the in the u.s u.s healthcare spend, is about 18 of our gross domestic product.

B

It's a huge amount to give you an idea of how that this compares to other western countries. um uh European health care costs are about nine percent, so stark stark differences in how much uh the us is spending on on health care, and so um what we've done is we've we've bolstered up our health care we've seen with both cyber security.

B

We've seen health care costs rising, and we know that the model today that we have is likely not sustainable, for you know the the generations to come, and so, as we look at this, we've been partnering with many open source, um open source partners, so everyone from spiffy to stiffy to spire. If you saw the last uh cncf um presentation, we were talking about it, there we're also in our world we're using oppa, nsm and then key cloak to to bring together a zero trust environment, uh an ecosystem, the the.

B

Why is really driven around, we believe um healthcare and the cost of healthcare shouldn't continue to rise. We believe that the the data should be um should be, should be shared and should be in places where it doesn't take. You know entire teams of hundreds and hundreds of people watching firewalls and watching security protocols to make sure that our our data is secure, that our transactions are have integrity to them.

B

So what we've done is we've started to work with various groups um and among this group, and what we'd like to do is we're we're working with cloud native engineers developers- and I know you- don't most people don't think of anthem as a place where technology is happening, so uh what better place to do it so we've we've collected, we've gathered so frederick and many other folks that are on frederick's team, as well as my team.

B

We've all gathered with the with the fundamental belief that we can shift the way that we do security, which then fundamentally shifts the way uh that we do and we conduct our business um for the better of the people. So with that this is this is our approach. This is what we're doing to set up a new way of doing uh doing business, and we believe that it's not just for our industry, but for several industries that are already uh working this jet. This direction that we're just partnering and moving along with them.

B

We want to be um not just at the forefront, but we want to be part of the change, that's happening so with that frederick I'll I'll turn it over to you and I'll be available for questions as we walk through this.

C

Great, so let's go ahead and get started.

C

um So let's get started with uh with a recap on some of uh some history, so in order to really understand zero trust, we have to look at where we're coming from and most uh security postures historically focus primarily on what we call perimeter defense. You have something that's untrusted that um that goes through some security thing, often a firewall into some trusted environment, and so very often uh it would look something like this. Where maybe you have the internet?

C

You have some dmz with some, uh where you have your application gateway and then another firewall again to your corporate network, and this also goes in the opposite direction. Maybe you want to connect from your corporate network to your internet. You have these multiple hops that you end up going through, and direct connections from the internet and the corporate network and back are implicitly denied. You have to go through the uh through the firewalls there's a variation on this.

C

This is a lower cost, uh maybe simpler to manage version of that where you have a single firewall and you have your dmz and since you're you're hop closer to the internet and corporate networks. So a little bit easier to set up a little bit less, uh I'm not gonna, say it's less secure, but uh if, but you have less places where you can defend, uh but also still perimeter defense.

C

uh We also have this in kubernetes, and so, when you connect in with client, you have some ingress controller that then forwards you into your pods, so once you're in the uh into the pod network, then there there are ingress and egress controllers you can put on which help tighten it down, but you're still very much in a uh in a perimeter. Defense model um multi-site same another, another variation.

C

We have a trusted, um a trusted network and another trusted network, and we have nodes in one side and services on the other that the nodes want to connect to. So you.

A

C

See this building on the previous set of defenses that we that we have and so, however, the details matter and uh the reason I'm getting into the details is because um it in order to produce a good security posture, a good security paradigm. It's not just locking everything down, because you can lock down things very very easily. The problem is that you have to lock them down while still providing your service, and they have the capability to scale that across a large number of systems, so you're focusing on a single computer yeah.

C

You can do a lot of things lock it down. I touch techniques and so on. But when you're starting to say, hey, we have an entire company, do we need a lockdown or maybe you're, working with fortune 500s and a lot of red hat? Customers can appreciate this, where uh you have thousands uh and that tens of thousands hundred thousand systems and these details, the security system, has to work in such a way that it integrates with such, and so in this scenario, going back to this example. We have uh two trusted networks.

C

There's a public set of ips, some private set of ips and a very common thing is to say: hey: let's go ahead and allow outbound uh from this network over to um to the set of ip addresses or and allow from this ip address to these particular services.

C

And so we have some form of network address translation, that's going on here, and those few questions then start to pop up like how what if I want to differentiate between node one and node two, you know I'm turning them all into one address in here uh when I do my network app translation or what, if I want to connect to uh to more than uh to multiple services in in here, which is a little bit of an easier use case, but it's still something that discoverability is still important on the server side.

C

You also have questions like how do you trust a client, and how do you expose your services out to others, and how do you differentiate between node, one and node two, if it matters and there's also a whole lot of other set of questions about?

C

How do you populate and rotate your certificates uh over time in the ways that these nodes over here are able to to connect in when you start to to bring in encryption, and uh one potential answer to some of these questions is well, let's go ahead and bring the vpn in, and so what we'll do is we'll say this.

C

This entire uh trusted network in 1i216a10 will trust and connect to this, and vice versa, and what they will do is they will inject routes so that they're capable of communicating with each other and when you start adding in the third one you end up with this mesh type thing uh you can sometimes reduce down. That's go down to a like a spoken hub model, but in in from a mental perspective.

C

This is what people tend to think of um a little a little bit of an aside, so the we're gonna deal primarily with layer, three vpns, and why layer three rather than layer, two, uh it all comes down to scalability um I'll, publish these slides earlier. I don't wanna go too much into this, um but basically layer, two tunnels are difficult to scale uh their layer. Three is built on ip addresses which is designed to scale and we use that to scale the internet.

C

um But let's talk about scaling subnets. So when we look at scaling subnets, uh when you have two networks, you're looking at one possible connection, three networks, three four networks: you're looking at six connections or six subnets, you end up having to deconflict and so over time.

C

What you have to do is you have to make sure that these all these subnets that you're hooking up to each other in these private networks, either you've, uh deconflicted them or you're pre you're, producing some form of net network average translation in a very careful way in order to make sure that they don't conflict with each other. So this becomes a very global problem very quickly and so um in terms of handling the conflict. You end up with very careful planning, but maybe maybe you've done a lot of planning, and maybe you didn't.

C

Maybe you didn't expect the type of growth you were getting or maybe you had. Maybe you you brought on a new partner or a new customer. Your partner, who has a larger network than you, were expecting or has requirements that they conflict with you.

C

um What happens when things begin to break you, you end up bringing in your your firewall experts, uh things like checkpoints um uh or cisco, firewall or so on, and you end up building up an it army in order to, in order to start handling all of these edge, perimeters and handling all of the network address uh the network, access control list, and so maintaining that list becomes uh very, very expensive, very quickly.

C

So back to the original question, then how does all of this stuff that I describe end up? Improving your security, because you end up with a lot of complexity.

C

um You end up with fragile configurations, and you also have entire sections around observability and debugging in terms of how do you, how do you deal with that uh and also we're still dealing with the assumption that the attack vector is coming from the outside and, I would say out of the entire set of slides- that's the most important line right there.

C

So, in other words, we are defending our infrastructure with 11th century technologies. What if the attack comes from in here so back to perimeter defense? This is the this is the world that many of us currently live in and what we are pushing towards is a zero trust, environment and I'll leave it here for a brief moment.

C

So what we're doing is we're saying the network is untrusted. It is no longer the the core trusted thing that you have and when you stop, when you stop trusting the network, that means you have to shift the trust somewhere else, and so where we are shifting. This trust is to the workloads themselves, we're saying we can secure the workloads and we can then develop secure connections between the workloads now to be clear. Just because I say that this network is untrusted does not mean that it is not private, it could still be private.

C

You still have layers of defense that you're building around, but if someone were to breach your network, you have other things to to mitigate and the the secure connection could still be a vpn but you're not relying on the security of that vpn for the majority of your security, and so when it comes to to untrusted uh networks. Please don't think that I'm that I'm saying everything has to be public, although if you want a really good limitless test as to how well you're doing you can play this as a thought experiment.

C

What if I take my network and make it fully routable on the internet? Like will I be able to sleep at night and if the answer is yes, then that means you're, probably doing it uh right with uh with zero trust, maybe you're doing it right so, um and so the idea is that if your attacker gets in here, then they're not.

A

C

To connect to other things, simply by being a member of that network, so the question then becomes: how do we achieve this? So we'll start with establishing a trust domain, we will attest the workloads. We will establish policy um between the workloads and we will also show an example with establishing trust between organizations, so the trust domains.

C

You know we're going to use the examples from spiffy inspire they're, both cncf projects um and spiffy is a specification on how to on how to get work, identity to workloads and how to rotate that identity, that the certificates that they're that they're assigned and do so in hierarchy in a hierarchical manner. So we start with the ca which is based on the x59 infrastructure, and what we'll do is we attest the uh the applications underneath of them and, of course there may be multiple layers in here.

C

You might a test of a sub organization which then has a cluster which then attests your application gateway, but the mental model. Think of it like this, you have a tested, a workload from some common from some common route and then once we've established that that identity and they've received x509 certificates on each one, we can then create policy.

C

So this policy is uh this is a modified example from open policy agent, um and so the important things in this scenario is: where is the uh is the we we consume, the the x 509 certificate from oppa itself and they call them messages. We can. We can consume the yesterday, the identity from it, and so once we have the identity in here we can check. Is this id equal to this, and so we say in the storage, api oppa would be sitting somewhere here or proxy right outside of it.

C

Is it receiving a request from this workload here? So if it receives something that does not have this identity and a certificate, then it will, uh it will reject that.

C

So, once you have that policy set up, then what we can do is we can is.

C

We can have a second organization that follows the same set of patterns and they test each other and if we want, if we want to trust who enter the two workloads together, what we do is we trust the two organizations at the top, the two roots, trust each other and say I I org one says I trust org two will attest properly and vice versa, and you also scope it down to their their domains, and so this means that when the front end app ap when the front end app server tries to connect to the storage api, it knows exactly who it's connecting to, and vice versa knows where the connection is coming from using mutual tls, which is available in tls, 1.3 and there's libraries to make this easy as well, and you can also use envoy to list things that are not aware of these type of concepts and then from the policy perspective.

C

What we're doing is we're, saying, allow connections to and we're setting a destination, we'll we'll match on the destination of where it's connecting to, and this site says, allow connections from and we specify the source destination from where it's coming from.

C

So this would allow those connections to persist and to run whatever set of http verbs that uh that you're allowed to do, and so let's drive this down a little bit deeper though so, when you're connecting two organizations together, you cannot assume that the underlying network exists, and so you have in this scenario, we have a character named sarah who's, trying to connect to secure corporate intranet using her application.

C

So just because you have the policy set up and the networks and your your identities are set up, doesn't necessarily mean you have a path to actually connect so you're. So you talk to your ops, people and the ops people say well in order to connect, we really need a firewall, an intrusion detection system and then this vpn.

C

So so, in that scenario, you have to think well, how do I actually go about doing all of this all this stuff, and so what we're pushing towards is uh in the network service mesh side, we're saying well, let's separate out the data plane, which is the things that uh that sarah wants for her application and let's put a northbound control plane over this uh over these elements, and so you can think of this similar to uh to an sdn in some ways.

C

uh The primary difference with the nsm is that it's it's more like an sdm for stms. It it'll talk to your two kubernetes and it'll talk to your checkpoint, firewall or whatever you're, using talk, talk to your intrusion detection system and talk with your vpn point and make sure that it passes information in and out. But I will won't go too deep into that.

C

I have a lot of other material that um that I can point you all towards for the internals as to how that happens, just the important part is think of it as a control, plane and a data plane, and when these connect with each other and communicate with each other, then that allows us to configure the things within the data plane.

C

um So what we do is we actually give each of these an identity. So your firewall intrusion, detection system vpn, they all get spiffy x509 identities, and so this allows us to know that when the pod is is negotiating with the firewall, it knows that it is talking to the firewall using a zerotrust model and simultaneously through the chain. So this allows us to build one once we have this identity that actually allows us to build policy. We can say this pod is only allowed to talk to this firewall.

C

This fire was only allowed to talk to the pod and the intrusion detection system and vice versa, all the way up until whatever is terminating that uh that connection for you. It allows us to build paths through that that uh that translate into your into your data plane using spiffy and oppa as those uh as those things so in in essence, we want something that looks like this.

C

We want sarah's app, it gets wrapped into some vpn gateway goes over some cloud, some internet to your vpn concentrator to the other side in your api um and so pushing those up the same concept as we've had in the application side. We have the same set of attestations on each side that occur and we have the trust that is put on the top oppa implements policies at each of the relevant locations and a side effect of this is that there's no more application.

C

Specific network acts control list in this, in the way that you traditionally do it. So this one, because we're saying this, this application is allowed through some path to to communicate with this api, using open policy agent and vice versa, so we've actually lifted that into something that is more declarative, something that's human readable uh that does not allow. That does not require you to put down this. Is this the the ipaddress of the api server?

C

This is the this is the ip address of where the vpn gateway is coming from, because we're now relying on cryptographic identity to do that, rather than rather than ip.

C

So one important uh detail to unify this is that these identities are can be cross-cutting. We can say this: identity is the same one that is used by your app, which is used by your service mesh, which is used by your pod infrastructure, kubernetes and eventually we'll get it down to the hardware tpm so that we can say this hardware tpm has came. This hardware came from a location that you have authorized that you've uh that you have yourself deployed.

C

So even if a rogue hardware comes in and managed to replicate your stack, even if it managed to gain access to some of your identity infrastructure, the fact that it doesn't have the right hardware tpm means that it would not be able to to fulfill the attestation when what it asks for what its identity is, and so we also have identity in the infrastructure and the other side has it's a set of identity.

C

As well so we have this cross-cutting identity for for each of these, and so, when we put it all together, we have the status quo, which is the internet. You have uh some client connecting in um and connecting into your application database.

C

I apologize like this is a slightly different. This is a slightly different use case uh from what we were showing before, as the previous one showed uh it being in a private network. I wanted to show off one more one more use case.

C

So, as we put it together, we have a client goes into connects into your your firewall or so on application to your database, and so the question that I would like to ask people in a scenario to ask yourself is: if you have an attacker here, how much access does this attacker have and there's been a number of very high profile uh break-ins, where the attacker got in through uh jakarta, struts or other similar systems that were unpatched or even worse, zero days once they got in here, they were able to do scans on the network and connect to the database or even connect to a database that had already had a connection to and start just asking queries that that are above and beyond what it should have been asking for and um when we start driving this boards um towards the zero trust model, the one uh one modification we can make is we could say: well, we have a client that we don't really know on the outside, but we trust that if the user has logged in properly they'll have a jwt id.

C

That is that is cryptographic that when we pass it into the firewall to the application server, this application server already has a zero trust chain between your application, server, your database server and database, and so we're already saying that the source and destinations for each of these are are have been established properly, and so that limits the the ability for this attacker to connect to some random database. That's on the outside, and it's that that's in the outside of this chain, and then we can further scope it down and say.

C

Well, we have a cryptographic token that comes from from the client and let's pass that in from we receive it on the application server or we've tested it at the uh in the inbound uh api server before it hits the application server, and so what? If we were to pass that on to the to the database? Server george with your database proxy and say well in order to unlock this api right here, this database server it not only has to come from app server, but must have a valid jwt id that we can then check.

C

Does this id here in the http request path actually match what is in the jwt id, but do this not at the edge of your infrastructure coming in, but actually do it further down and every time it passes through your infrastructure, it'll check. This is jwt match. This is jwt match, and so this means that you have that you have two conditions that you have to pass. You have to pass. Where is it coming from application server in this scenario and who authorized this request?

C

Well, the client did from uh from a client who was authenticated and an important aspect of that is. That means your violations in policy. uh If you have a violation in policy that usually means one of two things: either you have an active attack going on or you have a programmer who made a mistake that is causing your violation, a violation in the policy, either scenario you have to get somebody to look at it and to interview with the problem.

C

um So where does that jwt come from in in the open source uh in the open source path? We bring it in using something like key cloak, so key cloak communicates with our identity provider and that identity provider you notice it is not part of this infrastructure- is actually something that is managed uh separately in this example.

C

It could be in here it could be part of your your application infrastructure, but uh it it may also be a good practice to actually separate that out, so that, even if these continue, if these happen to get compromised, uh that your identity still uh still has some defense in it um and or some increased defense I'll say because you already have the zero trust model that's in there, and so this uh so the user is logged in they receive jwt and they receive it from clicko.

C

So key cloak itself allows you to have single sign-on ldap uh horizontally scalable, it's you're able to log in with oidc oauth2 sample social networks. It was originally created by red hat and I believe it's been donated to the cncf. Perhaps somebody can help me out there after we're done with the uh the slides, uh but the important part in this scenario is that it has the relevant things in order to allow us to integrate with um with identity providers.

C

In the back end, give us a uniform uh give us a uniform uh interface for that uh handle the login and receiving and receiving, and transferring the gwt, which then gets passed into your infrastructure, and so this this allows us to lift that zero trust even up to uh to trying to defend the uh the uh up to the up to the client level, even though there's limitations on what you do on the client. At this point, uh with those uh we have some uh some locations that you can go.

C

Look at for some of this network service, smash, spiffy, open policy, asian key cloak, uh envoy and kubernetes. uh A couple interesting areas where some interesting work has been done in here is, of course, network service. Mesh is continuing to build out the stuff at the layer, 2 and layer 3 level uh spiffy is working on something called transitive identity, so transitive identity is imagine this client that's. This is like a pseudo transitive identity that I described. We passed the gwt down in order to pass that through, but you think of transitive identity.

C

As you imagine, the client itself having an x509 certificate that instead of receiving gwt it receives a certificate that certificate has been designed in a way that it can say application server. I'm going to give you the authority to perform some action on my behalf to to the underlying infrastructure and so sort of like uh imagine. You want to talk with a um like. You want to have a lawyer, do something on your behalf or or and so you're.

C

So you give them uh permission through some legal agreement to do something on your behalf through some transitive uh uh authorization, beginning with transit identity of something, um and uh with that I think we have a little bit of time for our questions.

A

We we certainly do um and I'm going to because luca asked a whole bunch of questions, I'm going to try and unmute him, and let him just ask his questions directly here.

A

There you go luca hit, it.

D

A

D

B

D

Yeah, so um thank you for the presentation. First of all, it was really nice and uh really interesting. um I myself uh work as a pollution architect in api management, and I touched on some of these topics, so I find it particularly relevant.

D

um So when you were explaining the zero thrust scenario, um I think you were mentioning the fact that vpn and ids are still uh relevant, but uh do I understand this right that, in in this case, you would basically just need to configure uh them to connect entire networks without worrying about subnets and acls?

D

Is that a correct assumption.

C

So that's that's a really good set of uh observations and a good set of a good set of questions, and so in terms of intrusion, detection systems. um I put that in there because some of the companies that I've worked with are they their current policy is that they have an intrusion detection system.

C

That's in there, uh in fact, if in the most basic level of intrusion detection system, I personally don't think it's going to buy you that much uh some more advanced versions of these things could maybe uh help help you out a bit where, if you had something that was capable of of analyzing http requests to come in and look for anomalies in the type of requests that are being made or uh like.

C

Then you things that are there are more in band or that are working at the l7 layer, and then I think that we can gain some a significant benefit in in the model that I showed with uh with those type of um with those type of things, but they the traditional l2 and l3 uh model. That's that's! On there uh it's there.

C

There may be some areas where it's still useful, but it's, but because you're you're, limiting where your word receives messages from at such a fundamental level um that it it does, reduce the total amount of value that you would get out of an ids. Even though there's still value there.

D

Okay and the other question I had was when you were showing the jwt validation.

D

I think this is that the opa is getting a little bit into the api management field right because, typically, that's the one of the tasks that the api gateway can do. So I don't know if you guys had already also an api management solution and managed to use that as well or.

C

uh Let me go a little bit more into this particular path, and so uh open policy agent is just uh all of all. It really is, is you think of it like a as its own server, and you can embed it into an application? If you choose to do so, you send it, uh you send it a string of json.

C

It responds back with a successor failure and maybe an explanation why, depending on how you've configured it and so it's capable of consuming uh within those strings or within those those fields, you may pass in uh things like the headers or you might pass in uh some, the the svids or jwt tokens and so on. So it's not actually sitting as a as it's not sitting here in this area, controlling the axes in between or so on.

C

This is actually something that we would want to rely on, something like enway, proxy and all the way proxy has the community to inspect and to perform that. um Interestingly, a lot of service meshes can help there, because they already have uh connections and the capability to configure enway the there are still some gaps, though, uh because many of the service meshes.

C

Don't implement the spiffy protocol or if they do implement speedy protocol they've implemented in a very opinionated way that may not make it as easy to integrate with other systems, and so one of the things that I've been uh looking at is working with some of the organizations.

C

uh For example, I've been working with uh I've been having some initial conversations with uh some of the individuals over at uh cong as an example, in collaboration with uh with network service mesh, because we have a poor committer, who is both a part of kong and part of network service mesh and we've been discussing about.

C

Since we use uh spiffy inspire within uh network service mesh already, it would be good to be able to get that identity from spire in both nsm, which we already have built out and also be able to get that identity from uh push into kong. So that way, you can reuse those identities at that location rather than having two different identity solutions that we have to work out, how to make them uh play nicely with each other, and so um istio also has a spiffy support through a product called citadel.

C

I have not looked at citadel, so I don't know how well it integrates with uh like. If I were to stick a if I were to expire on top, could I have spire control citadel, or can I do nested citadels? uh Can I get citadel to to work standalone? If I don't want to bring this you and I want to start bringing in things like like these patterns that I showed off you. Could you could easily run them in in openstack or or other environments with given a little bit of development towards that direction?

C

And so um and that's part of the reason I was looking at something like spire for this was because fire runs as a standalone. It does not really acquire kubernetes, but it integrates very well with kubernetes right.

D

Right uh just one last question: um you were showing at the beginning uh when the you were explaining the setup for zero trust, the trust between org one and org two. I remember right.

C

D

So is this like a a typical scenario or in in practice you would have actually some more, maybe just a department or even development group trust, another development group to make it more strict in terms of validation, so that nobody can actually like reuse the certificate and actually access uh somebody's else back end. Or would you limit that in the policy with opa.

C

Okay, so there's a there's a couple things towards this. So what the way spiffy works is these certificates are very short-lived, so the default out of the box is that the the northbound cas are rotated once a day. The southbound workloads are rotated once uh once an hour, either actually rotate every 30 minutes, but they're they have lifetimes of an hour and they're rotated 12 hours but lifetimes of a day. So someone were to compromise the system and extract the certificate out. They have a very short window to perform their their attack.

C

So that's the first mitigation. The second mitigation towards this is the way that uh spiffy, uh that's that spiffy actually does the attestations. So the the applications go up the chain until you get something that can perform the actual approval and so you're able to scope what type of approval.

C

So if I have, let's say that I had a third system under org, one that was like uh payments uh payments api and I tried to to attest that through the front, end app server path and say well, when I'm establishing this connection they're not supposed to be in the same cluster, then uh they wouldn't. It would not be allowed to to perform that at the station because it would not meet the uh the requirements for that particular system.

C

So we can scope this down so that this cluster, this sub oregon cluster, is only able to receive certificates that are relevant to its uh to its needs and prevent it from testing other things.

C

The reason that we want to trust them at the very top with the top level cas in this scenario is that we also want it's it's a trade-off to it so to some degree, where you want to minimize the uh the total number of uh of connections that, like we don't want to say here, like let's send the front-end app server every time we rotate certificates here, let's send that all over to second organization and vice versa, uh and so by establishing at the top, you reduce the total quantity of communications on there, you're not telling them hey.

C

I have an organization, here's the structure of my organization, you're.

D

C

Telling them enough information, so they know who they they're allowed to connect from, and the other thing towards. That is that it also gives us a single break: the glass location, because, if org one and or two decide decided they don't want to communicate with each other. We can literally destroy the trust here and that will propagate through your system um relatively quickly, because when it asks for the next for the next bundle, then that'll um that'll end up removing the the bundle on there.

C

And if you compare that to stata to the status quo of most organizations in this particular area, uh trying to perform that style of rotation uh without spiffy, uh it can often take weeks or months if you need to if you need to go and configure and redeploy a bunch of software.

C

So it gives us that dynamicness that helps mitigate some of those. uh Some of those concerns that you're describing.

D

All right, so the just one last question: given the explanation about spiffy, um so the whole rotation is managed by spf itself, because I know that certificate management is always a headache in general. So.

C

Yeah spiffy is is a specification. It's part of the cncf, it's a cncf project now, but it is a specification designed specifically for or doing the attestation of workloads and then rotating this over time. So it's uh it's! uh It's it's a grpc based api, uh it's very easy to to build against, but it's designed specifically for uh for solving that uh that rotation problem over over a large organization.

D

Okay, thank you sure.

C

A

So I I have a just if you're done luca, I have a question a lot of this um zero trust model is really about communication between services and technical um between your internal systems and and services, but maybe for for bobby.

A

How do you communicate this new model to your internal compliance, audit, I.t people, so that they trust you that you're implementing this correctly, I mean that's, got to be a big issue at a company like anthem, um especially after the breach.

A

How did that conversation go down, and how do you do that with like multiple acronyms, throwing that all at the compliance officers and your it audit people? How does that fall out at anthem.

B

That's a very insightful question: um it has not been uh it's not been as straightforward as it would seem, but the the way we've started it is, um and just just to be clear today, our chief information security officer, our ciso, is one of our key sponsors for making this change.

B

So that was something frederick and I took on day one as we were talking about this and socializing it so you're right, it's a lot of acronym soup and it's a lot of new. It's uh very new models for um traditional sorts of infrastructure. So what what we've started with is doing this outside of the traditional systems that are up and running and that are really running today. So we're going to start first with proving this out in systems that are maybe, let's call them less core. If they go down, things are okay.

B

If they get, they get hacked or breached it's okay, um as we prove it out, um we'll see, and the the ciso is working with us team working with us to watch how it works to do. Pen, testing, uh in fact we're making our source code public we and they were their eyes, were like kind of we're in horror like are you? Are you serious? Yes, no! No! We we want you to know where every hole.

B

Every weakness is we we want you we'll even make it public to the people you pay to come in and and uh your white hackers, just white hat hackers just come on in and do this um because we're we want to prove this out and we might as well prove it out with in the most dramatic and open and transparent fashion, because this is what this technology is about.

B

So that's how we've started um in say: give us give us about six months and uh frederick, and I will have our internal prototypes, not prototypes actual, like functioning apps up and running we're past the prototyping stage um and then anthem is uh in general, has not been a fast follower, but as we're looking to move more of our core on-prem type applications into cloud, they will just come on to our stack so we're even building our entire stack to that core applications.

B

They can bring their old security, um they can bring their old security uh paradigms with them, but they will be on this new stack, so they'll just be moving to the new stack and one of the issues diane that people run into the insurance uh or in the old, traditional brick and mortar business that evolved into using technology um is if there are no customers, you it's hard to get people to change, well, they're, actually bringing their customers with them.

B

So it's not a migration from an old app to a new app they'd, be just moving their entire customer base onto the new platform, with the with the new, uh with the new apps that, with the apps that they're moving from on-prem into cloud native wow, so yeah, it's a it's a big strategy. It takes a lot of time and effort a lot of people to keep. Let's just call it kissing, hands and shaking sorry shaking hands and kissing babies.

B

A lot of that as soon as kovids kovid's uh equipped out, but just keeping the connections warm, reminding them what we're doing um and keeping this to the forefront of their their mind. But they also have they'll. We also have some headwind.headwinds.

B

We have some some wins in our sail in that it just costs a lot to do with the way we're doing it today and it's not sustainable.

B

Those are all helpful things that work in our favor and so we've we've been welcomed with uh open arms.

A

All right well: well, that's good news, and in six months we're going to have you back um to talk about where you are in production, um readiness and and whatever lessons you've learned over the past six months, um and so I I'm looking well lead you kind of had one. I think you had one other question and I'll see if um oh eric eric is that has just asked um a question a little bit ago. Does this rotation itself represent an attack surface?

A

um Let me just see if.

C

Not it's a great question. I would argue that anything that you can communicate with represents part of your attack surface. uh That includes the rotations, which means you have to have the proper security audits done on it, make sure that uh you're securing those endpoints properly.

C

So uh I would argue yes, that is part of your part of your attack surface and the same way that your firewall itself can be, uh or your virus scanner there's uh scenarios where fire scanners have had bugs in them that have allowed people to compromise systems, uh and so uh please, please treat it. I in fact I insist you treat it as part of your attack surface and that you, you analyze it to make sure that it's being uh secured properly and that problems with uh within it uh are also are also mitigated.

C

um Spiffy and spire were designed uh with that in mind uh in terms of reducing the total quantity of privileges towards the southbound similar to uh kubernetes. Does this with with pods, where it can expose portions of the api out, uh but it will limit what uh what things in the southbound can do based upon its uh its capabilities, and so uh but yes, in short, that that does need to be. uh That does need to be considered perfect.

A

Well, we're almost out of time- um and this is all new to me and new- to a lot of the people who are on this call, so we're definitely going to have to do some more follow-up conversations around this so that we can all learn from your experiences and your expertise. So thank you very much everybody for for sharing this.

A

We really hope that you get something from this conversation I'll be posting it on youtube and I'll get the slides from frederick and post them as well on the openshift.com blog and please reach out to both frederick and bobby um the slot. If you throw up that very first slide, we didn't put your emails up there good on that.

A

That's a that's a trust surface that we didn't want to touch on or something no spam, but um we'll we'll figure out how to get you guys connected um so as well is, if you come into the slack um for openshift commons, if you're not there, yet let me know and I'll add you in, um and we will continue this conversation um online. So thanks again, frederick for reaching out and bobby for taking the time today um and as always for being participants in the commons. We truly appreciate that.

A

So thank you all very much and take care.

C

Fantastic. Thank you very much. Thank you. Take care.

D

D