Cloud Native Computing Foundation Notary Project, 24 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CNCF Notary v2 Key Management 2020-07-24

Description

CNCF Notary v2 Key Management 2020-07-24

A

A

Hi how's everyone doing.

B

B

Good, how are you.

C

A

All right, uh marina, do you know if uh justin is going to be joining today.

B

um I think he was planning to, but I don't know.

A

B

D

uh Sorry for the delay, I had an audio problem and had to restart.

D

Who else are we expecting on in this.

A

Meeting, uh I'm not sure, uh let's get, let's give it a couple of minutes and see if else.

A

A

Just gonna go set up the.

A

A

I think we can get started. I don't think anyone else is joining today. um The uh what I kind of wanted to get done today was go over the pull request and finalize that um and uh marco, I think, you've already taken a look at it. um Justin have you had a chance to uh review it and uh come up with some comments or feedback.

D

Is this what uh I was looking at last week or I just want to make sure I'm talking about the same thing? Can you just post the link in chat, so I can.

A

Yep I'll put the link in chat. It's the same one. I added in some of the uh uh the reasoning uh like we discussed last week uh so to clarify. So let me put this in.

D

D

Thanks, I'm playing it up.

D

D

Yeah, I do have a couple of questions I have when I'm looking through this.

D

So one thing: um okay, so you talk about the publisher.

D

And the deployer as different people, so the the deployer as I assume the person who's actually just sort of consuming. The containers like if you.

B

Know what I mean like.

D

Yeah, okay and the publisher is the person who's actually creating the containers um and then there's another party here that I think is implicit, which is sort of the person who's running the registry.

D

um Yeah- and I don't know how relevant this is, but I'm just curious about one, a couple of things that come up when I start to think through them and what they are supposed to be trusted to do and what they're supposed to not be trusted to do more. Specifically, because I think in your system, you want the the publisher like someone who is a publisher, which I noticed, do you have like the publisher, admin and builder and signer?

D

um You want them to do things like presumably control which which images are trusted.

D

um Is that correct.

A

um So um what do you mean by which images are trusted right uh in the sense that there's two there's two different definitions of trust here right uh one, the employer is saying here are the different publishers. I trust if anything is signed by them.

A

I will trust that software and then there's the other component of trust where a publisher is saying here are the uh here: are the containers or things that I have signed that I'm saying you can trust that having come from me right um so uh which one of those are you are you referring to here.

D

um I I think they're both very related, because I think the first one deals with who you trust and the second one deals with. As I understand what you're saying the second part deals with what the people, who you trust, say right um and so in.

D

In the end, I think the like, I think, the design you're in part arguing for which I think is which, as I as I understand it, and is that the publisher, like some set of people who are the publishers, are the people that um the deployer who has picked those publishers has enabled to decide what um containers should be installed when, when you know when the deployer goes to make a request.

B

D

D

um Is there a sense where it's not true.

C

Go ahead so, from my perspective, I think we we should uh split these scenarios a bit from two to two angles, so you have the consumer part where you would like to know. Okay, what are designed images and and what are the keys, which I trust.

C

So I think that somewhere covered using these root keys, which you can somehow white list uh on on your uh registry, but, on the other hand, uh that's the topic we at philips are trying to resolve with notary v1 today is how do we manage which developer or which ci job is able to sign an image at which point in time?

C

So let's say today I have a team of five people. One guy leaves the company the other one moves to another department.

C

uh In that case, um we want to be uh have the team be able to self-manage the delegations for the given target repositories, but on the other hand, we also want to have control on um creating target keys and removing target keys and and and doing things like key rotation and that's more or less a administrator kind of rule.

C

So let's say you start a new project.

C

The first thing you will do is request for a target signing key to this administrator team. They will create you this target key and on this target key. You can create um your delegations or remove the allegations or replace your key with with a new one.

C

So that kind of things and for this we we have this proof concept um justin you. You missed that uh as you learned in in that meeting where I gave the demo, but but it's uh um available open source, um it comes with it with a small web interface to to manage these kind of things, but I think in auto revv2. um As far as I can see now in in the um design, this is not not covered.

A

Well, I think, actually, if you look at the hybrid signing use case and then the signing by build machines or dedicated machines, that does cover some of that right, we're talking about the administrator, creating a a root key and then delegating access um for uh the the entity, that's essentially being designer, whether that's a developer, whether that's a build machine to be able to actually get delegate keys that change from the root. um So that's something yeah. I think the coverage.

C

C

Personas, we could make them more explicitly or elaborate a bit more on what the persona does and what what it means to make them more explicit.

A

um I think that's that's uh uh I I can. I can take uh uh elaborating kind of like who can take on those different roles, but I still don't think we've answered justin's initial question, um which was around: um how are we determining which artifacts we are going to trust overall ray? uh I think justin here the way I was envisioning it is that you know you're essentially like you call that the deployer is trusting some entity to say these artifacts are trusted.

A

These artifacts are okay to run, and then it is up to the entity, that's being trusted, in this case the publisher, um to kind of manage uh how they say here are here's here's, the ones where we have signed, and we we. We want our sort of like consumers to say they can trust us, and then they have also mechanisms to say something that we have provided. Trust us before uh is no longer trusted right.

A

um I think at a very high level, that's kind of what what we kind of envisioned, um but um are there any gaps that you see there um that that that's uh uh that you want to address.

D

I mostly want, I think it would be helpful to be very explicit that this administrator party is not trusted like the registry party and to be clear about that part of the process, because actually, I think the biggest concern that that we have overall is that um that a party that has control over a registry can do things that are um extremely damaging and- and so um you know, I've heard some discussion and some you know like oh well.

D

Maybe the person goes in and clicks buttons in the registry, but doesn't sign anything, and this picks what this thing links to or something like that, and I think that's that.

D

Obviously um you know if what we're supposed to be having here is we're supposed to have the the publishers and the people at the organizations actually deciding what gets installed and and what's trustworthy, then than that those are kind of very different models than one where oh well, I just trust the registry, and- and you know I don't need to sign anything because it comes from the registry. The registry is a good guy which isn't isn't the model that steve or anybody's proposing. But I think there's flavors of that that creep in in places.

C

I I think, that's that's also what I meant with we. We should split more the the the role of the consumer and design also also in in the admin part you have the admin part where you trust keys, which are uh derived from a from a given root or a given intermediate, where you can white list. What what is the things?

C

We really trust so that that could be a subset of images from different registries um and, on the other hand, you have the part where you are managing, who is able to sign the images for a given repository.

A

Yeah, I think the part of the goal here was to completely remove the role of the registry in the trust components, and I think that can be addressed by adding in a registry persona that says that here are the places where the registry owner, if you will gets involved, what does their role mean? What do they add and where are they required?

A

um The way this was written right now. The only place where registries are getting involved is in validating uploads to the registry and saying that you know when we're getting artifacts coming in tied to a developer identity.

A

You can validate and upload to the registry whether this actually was signed by the developer in question or not or the publisher in question or not, but even then uh the when those container images are being pulled down by the deployer uh they're, essentially still able to revalidate that and that's happening independent of the registry right.

A

um So I think, if I add, in a registry, persona uh and kind of show the end-to-end workflow of, like you, know, a container being signed, uh getting uploaded and uh getting pulled out and deployed and show sort of like what role each persona has there. I think that might address some of the concerns I'm carrying, because I think that is missing and clarity from the uh the scenarios right now.

D

I I think that would be interesting. I think that would be helpful to see. um I have a couple of other little questions, but I don't want to kind of monopolize the the meeting here, but um what one.

A

D

Okay, like root, revocation, um which you say compromise a compromise route, should not be needed in process to designate itself as revoked.

D

D

Don't really understand this! Well, um I, how does okay so.

D

You've you describe a way this could be done, which basically means you have to have a way to touch all of the clients um in, like the second to last step.

D

So, basically, as I understand it, what you're really saying here is generate a new key assign new metadata and go to all your clients and tell them to update their key.

A

um Yeah, the part of where this comes in from is in notary view. One uh one of the big concerns is if your root is compromised, that root can itself then be used to rotate to a new version of the root, uh and so I think separating out like how you do root, revocation from sort of like essentially other revocation, uh was one of the areas that we wanted to address in arabi too right with roots.

A

It becomes tricky in the sense that there's really nothing else to sign up once your group is compromised, and at that point it's really just a root rotation and saying no longer trust. The older route is the information you're trying to propagate. At that point,.

D

So so this is very interesting to me because.

D

It's it's almost like the features is a bug, and I can't quite figure this out. Why.

D

So what you're saying is that you don't want the ability to rotate root and the reason why you don't want that ability is that the attacker can rotate the route, but if the attacker like okay, so I just I'm trying to walk through like a hypothetical situation here, that's okay!

D

So let's say the attacker needs to have two things: they both need to have the root key and they need to have access to be able to serve content, as as, though they are like a trusted source like a registry or something does that make sense so far.

D

Okay, so they have those two things. um Let's say they have those two things: okay, so now you have two possible designs. Design one is: um is the original tough uh notary, v1 design where they then can rotate the route? Okay? So then they rotate it to a new thing.

D

They've gone they've been able to compromise and do anything they want on on any of the parties that trust that root key. For that specific repository, which in notary v1, is a very local repository that only contains information.

D

um You know like it's: it's it's information for a small group of developers not like, like every repository that exists or something on docker hub; okay. So so that's scenario one so so they can take things over install what they want. But.

D

And you don't have a way to rotate, to get rid of that or to change that now. Let's suppose that we use this other design, where you don't have this root, this ability to do root rotation.

D

Now, if you don't have the ability to rotate root, but the attacker has the root keys and the attacker has taken over the registry, then the attacker can still have the party control like control and install any software. They want.

D

um And in both cases you still have to go to all the clients and touch them to recover from this.

D

So right, yep go ahead. No, I was just gonna say so I don't actually see what this what benefit this gives you.

A

So I think part of it is um the requirement to have to go to the clients to enforce that right. So with an automated rotation, I think there's a uh there's a potential that the root that's like.

A

If the root is being used to kind of, say, here's how we're rotating it, uh then the uh the attacker can change the route right and then, from that point forward, uh you're automatically locked up from rotating the route yourself, because you no longer have access to the new route that the attacker used to kind of, like the sign uh to do the rotation with right.

A

D

Case well sort of, but I just want to ask a clarifying thing here: I'm sorry to cut you off, um but in. If you don't have the ability to rotate, then you always have to touch the clients right.

D

So I I don't what, if you're always able to touch the clients? It isn't the do you see like I'm, not I'm not sure, I'm expressing my confusion well, but I'm think.

A

I think what I'm suggesting here is: we have a different process to touch the client right rather than having to go through the registry and rely on the route.

A

I think what what I'm proposing here is that the clients have a mechanism to understand what the routes are, and this is not using the the the keys that are being used for uh signing the containers and images right, so this would be kind of where I think the notary server morphs into. So, if you think of notary server as being like a repository of keys that are trusted for different entities, then you can essentially check the notary service to see what the latest set of trusted routes are.

A

This way, if someone wants to, like you know, say like here's, a new set of routes that we want to use, they can upload that to notary server, and you can your clients can check that to see what the new set of trusted groups are. So that allows you to do group rotation um without having to use the uh the root as sort of like a uh as an authentication mechanism. If you will.

D

So, okay, so you talked about the notary server doing this so or this notary service server. I want to make sure I'm saying this right: does it does it matter which way I name.

A

It yeah- I think this is this- is this- is something that we can look into more in the design, but essentially what I'm? What I think comes out of this as a requirement is, we need a component uh that is telling you what the latest routes to trust are for any publish for for a publisher um without having to uh use the keys themselves, as uh as as mechanisms for authentication.

D

But but how do you know who that party is, and why is that party trusted.

A

um Well, why would you plus, uh like you, you have to know who you are getting uh suffer from in the first place, right like when you're saying I trust this party, uh you know who it is that you're trusting right.

D

Are you talking about like in uh in like a tough repository or are you talking about in just in general,.

A

uh In saying more in general, so, like you know, if we, if you think about the concept of like a deployer saying which publishers do, I trust um you would verify who that publisher is- and uh you know where they're, where potentially to get their key information from right.

A

um So this could be sort of like you know if you're saying like, I trust, let's say uh I don't know, let's say, let's make up a company if I trust acme, um here's where I know acnes, like keys, are published from, uh and here's where I'm going to go to get that key information, because I trust acme as a publisher, then anything that I see signed with acme's key. I can verify that they change from their root.

D

Right but but the reason why the ca system, like does all this and works is, is because effectively a bunch of um rules and regulations were written around what it means to be a ca and how to do things and the ca system, because it has a lot of parties involved and has a lot of trust in those parties has actually been a pretty big example.

D

In many cases of of sort of, like the trust system, gone awry, so I, like you, know the fact that you have 500 certificates like for different cas. All completely trusted by your browser is is, is you know not a not a great situation given all the digi noter and other hacks that have happened in the past, but but how would you I like?

D

I think I need to digest this, but to me it feels like you're taking and you're you're, creating a new, very trusted party. That is the like. A third party. You you're, I think the way you're talking about this is this is some. Is this a server that everybody sets up individually? Every individual party sets up a server that has these root keys.

A

um I think that's one where we yeah, we can.

B

A

More into sort of like how that is set up right, like that, could be done in a combination of ways for um for publishers that have a strong notion of trust where it's kind of, like you know, hey like. I absolutely want to make sure that uh no one else is able to kind of compromise me like have like you know, like you know like.

A

If you have someone uh you could set up this yourself and then this information yourself uh on the flip side like if you would want to kind of go down the ca model and say, like you know, I don't want to run the server myself. You could potentially, like you know, designate a third party to establish and share that trust for you right. I think this really goes up to like each individual publisher determining what their trust boundary looks like.

A

um If they want to establish this trust themselves, um they can set up the service themselves versus if they want to designate some other party to vend this trust for them. um They could use a third party to do that right, but I think both models would potentially work there.

C

So so, based on this discussion, I I have some more questions.

C

So so let's say we have many notary servers, and many of these notary servers could be cloud could be private ones, and let's say I would like to run a private server in my close to network environment, but I still want to be able to trust some of the images from docker hub based on trusting a route or trusting uh some other level of certificate so shouldn't there be also some something to administer, uh how your server exchanges the root certificate information with other notary servers.

C

So you can basically build a swarm of servers where this information can be retrieved from so my client only needs to connect with my particular notary server um and then just propagates to two different servers to to retrieve the the information which is not stored locally. So I'm more talking about a distributed model where you see nowadays, a lot of things are moving into these. uh These distributed models where the information is not on one server but where it's just spread across uh the cloud or the internet.

C

B

C

Keep the ownership like okay, who is owning what and and who.

A

That presents some risk in the sense that if you have a distributed model of trust where this information is propagating through whenever you're doing a rotation, you also need to kind of make sure that uh it's going through in a reliable manner to all of the uh uh all of the distributed hosts and things like that. Right, yeah.

B

A

So I think, having a central server makes more sense in the sense in time in terms of making sure that, if you in in this in the regards secure rotation or vacation, there's a central authority that you can essentially go and verify that information from for air gapped environments right. I think this is one of that.

A

I have a section in the trust or configuration which we can expand on. um The idea, essentially, would be. You can periodically check uh if you want and pull in that information, um but you wouldn't get that sort of, like you know, immediate information, if, like a revocation or something happen, um it is, though, that, like you know, we could kind of come up with some kind of a notification mechanism that you could set up.

A

That tells you when to pull more information to your gap environments, but I don't think the distributed model of trust itself. It would end up becoming a lot more complicated to manage and make sure that uh all the uh all the information gets updated.

C

Yeah, so in that situation you would need a secure trusted, uh synchronization protocol or something like that, not sure how that would look like, but that that would be definitely something that that you would require. But um let's say, if we stick with with um with the central server, uh I foresee one wants one issue there, that's with some of these big enterprises, even even in some of our own products or things which, which are deployed at hospitals. We simply are not allowed to do a network connection outside of the trust network and stuff.

C

Like that, it's it sounds crazy, but still 2020. This is the case, and let's say these: these projects, some of them they are running docker containers, but the way they are running them. It's it's crazy.

C

Instead of depending on an external registry, they make zip archives of the docker image they make checksums for those things they transfer those zip files, including the checksums, to a system where someone manually verifies the checksums and then imports the docker images into the internal docker registry, simply because we are not allowed to push internally into their registry.

C

So those kind of use cases if if we would like to get those people on board, there needs to be some some kind of mechanism where you can still have your own server and still have a way to synchronize some some things and.

B

This is a very extreme.

C

Use case I I know, but I I'm not even sure if, if you would even like to support this kind of things, but um maybe that's something.

C

uh We should also give a thought because at the moment, with notary v1, what I see with most people is that they they find it difficult difficult to understand, difficult to manage um with all the constraints they are also getting from from clients uh like I just explained, but there are also use cases where, where it is, of course it's not applicable where it's, where it's a lot more straightforward and yeah more cloud agnostic.

A

So there is a workaround that would currently work in that sense that you don't necessarily need to go to a server to get the keys. You can also configure the keys yourself right so in an air gap environment. One of the things you could do is you could look at what keys are trusted and configure them yourself, but that is at a point uh sort of trust.

A

Right, like you have to go back and update that at some point I think in a true air gap: environment where you don't have that network connectivity you're going to need some manual process to upload that- um and I think that's that's- that at least the ability to kind of configure keys locally would give you that ability.

C

Should we describe that as a as a persona who, who needs to be able to do those kind of things, or is this really edge case.

C

A

A section uh I have a section for that in uh on line 71, I will expand that to kind of show some of these, because that was something that steve talked about as well, that we should kind of like clarify how air-gapped environments would build the trust and also the signing as well.

C

Yeah yeah so yeah from my perspective, I think it also makes sense if we make a more specific splits on the on the personas, because now we have publisher admins, but I think you can even divide those admins on on different use cases and maybe also different steps in the process.

C

So, let's say someone managing who is able to sign images uh could potentially be someone with a different world and than someone who is managing uh the the trust of uh certificates, roots and things like that.

C

I think making that a more explicit split uh probably also eases the discussion like uh who is doing what and which scenario. Would they do these kind of actions.

A

So the deployer in this case would be the person that's responsible for doing that. Let me look at the. um I think. Probably another role that we need to add in here then is the customer administrator.

A

um So this is necessarily like the same way: publishers fit into an admin and a signer uh deployer gets split into an admin, who's, uh configuring, the trust and the developer running the or a machine. That's actually pulling the containers and running them. So I think I can split deploy into two. So um if I'm carrying the feedback on the personas, the uh this really should have five personas going forward uh an admin and a uh assigner for publisher uh and for deployers.

A

Similarly, an admin and a deployer and then a role for the registry owner.

C

Yeah yeah: I think that that would allow bigger organizations to to scale the the work better and also you know, to to keep control on what is happening in in the.

B

Organization, I think, similarly, for that scaling property. It would be nice if the different personas can like delegate pieces of their role to other parties, especially in like a very large organization. So like you could have multiple people um in charge of a particular um persona like it doesn't have to actually be a person or even a single key or anything.

C

Yeah correct so so in the poc we built today, we were now thinking to just add some role based access control with with jwt tokens on the web interface part. But of course the best use case would be that that all of this is managed within turf itself.

C

So so it's not depending on this web overlay, which we now have as a poc, but that it just is embedded in the tough framework on, unlike who is able to to add new delegations and who is able to remove them or who is able to rotate the the certificates for for a given target.

C

If that could be embedded in tough indie signatures, then it's more watertight than having an overlay solution that that builds this. On top of it.

A

um I think those roles would make sense, but do we want to make that distinction now or do we want to work on a design and then figure out what the different roles are and come across? That? I think what I was attempting to do right now was at a very high level capture what the different use cases are.

A

So we can further blend it into the design, and I think that design would spell out what the different uh activities are and then I think we can go more into like what the role distinctions there need to be.

C

Yeah yeah definitely that.

B

Definitely makes sense, I think it's good to start with. You know, what's going to be done, and then we can figure out how to make sure that all the right entities are trusted to do those things.

C

A

Sense. Okay, so I think um uh what I'm going to do next step is uh um add in those additional personas. uh Add some more details to the trust for configuration, use cases uh but justin. I wanted to go back and I think uh we still um around sort of like the root kind of uh management piece. um We didn't quite really settle on an answer there. um What are sort of like some of the additional concerns that um you think we should debate.

D

um I'm still like I still when I try to kind of reason through some of this, especially around um around this, like additional party for root. Revocation, I still have a really hard time. um I think it's just.

D

I need to to go through the scenarios because, as I kind of map this out in my mind, it feels like this is just adding another party, um and- and I I I you know- I'm gonna probably go back through the recording for this and try to um understand better where you're coming from, but I I still have like from just a security trust standpoint. It it feels like this just makes this design weaker um than than having the root.

D

Do the rotation like like you're, adding in another party um and and I'll try to have a an easier way to reason about it, but like fundamentally, the way I the way I still see it is. Is that the the the trust like you you're you're, just adding a party that you're that you're relying on even more um and you already were in trouble if the route was compromised?

D

But now, if this other party that serves as the central like, uh I don't know what you want to call them, but the uh is that that's not the trust store right or is it.

A

No, it isn't. uh The truster essentially is the deployer configuring which keys or which entities they trust. So that's not the truster.

A

um I think, let's do this, um I'm going to clean up some of the language here if you want to take a look at it next week and then uh come up on friday with sort of like uh where you see sort of like the concerns being. Maybe we can address this in the next meeting.

C

A

C

Maybe it also makes sense if we add a small diagram on how these things would connect on on a high level to to get a better understanding to make sure that we are on the same page.

A

I can do that. I can throw in a diagram this weekend, um so uh justin I'll I'll I'll. Add in some of the comments we discussed today about the personas I'll add in some diagrams.

A

uh If next week you can take a look and uh then I think we can have a debate around uh what the root management means and we can uh probably settle that next.

C

And- and maybe another thing is, do we need to decide that, basically, in this phase how the root management looks like or is it just enough to say, okay, we need some way of managing route and whether that will be done via um rotating or another way. That's probably also more in detail design right.

A

um I think where it sits is uh something we will want to cover now, um because it does kind of, I think, play into like how routes are getting rotated, how, when you're doing key management where that's plugging into and if we are essentially coming up with a requirement for publishers to manage a service.

A

That's something I think we should close in with the larger audience as well and say we're okay, moving.

C

A

C

A

C

A

And then we can go back to the larger audience and see what what the consensus there is.

C

A

Okay, um was there anything else I didn't have any other. I think I've gotten some good feedback today as well for the next round of uh changes on this stock.

D

Yeah, I still need some some time. I'm sorry! I didn't um prepare well enough for this meeting. I still need some time to digest this, but I'll have more feedback. uh Next week too, okay sounds.

A

Good uh marco marina, thomas.

C

Sorry, can you repeat that.

A

uh It's asking: did you have any other comments.

C

uh No not from my site.

B

um I don't think so. I think I need to also read over this more carefully because I skimmed it before this, but I should read over it and if I have anything else, I'll just leave comments.

A

Okay, um I'll go ahead and publish another uh update by monday morning, uh and then we can, um if uh once you guys, get a chance to review. If you put in comments, I can also start looking at it for the friday meeting, uh but next friday, I'd like to close this and uh the monday after just shares what we've agreed upon with the larger audience, um then we can start like focusing on the design aspect of this.

B

Yeah sounds good.

A

All right, uh thank you all. um I think.

C

D

Yeah, thank you. Bye.

C

C