Ortelius Open Source Architecture, 10 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ortelius and RedHat Emporous Jan 10, 2023 integration discussion

Description

A continued conversation with the RedHat Emporous team on integrating Ortelius with the Emporous DB.

A

B

Yeah, we see your browser now.

A

C

So there's nothing in here that I'm going to show that's like, even though I I, it's my slide. Deck and and uh Andrew's slide deck. Who is he wasn't on the call uh last time so Andrew Andrew blocks uh uh with us today he's he helped me create this deck uh Andrew? Did you we've done introductions last time you wanna say something about yourself.

C

Or not, that's good, too. Okay, um let's see.

D

So I could not get myself off new guys, I'm. So sorry, I said Zoom was being said. Hey everyone, sorry about that. My name is Andy block I'm, a distinguist architect on the red hat side, I've been working with Jay to help kind of bring more visibility, alignment and an understanding of what Emporia is looking to do within red hat and in the community. So I've been working on efforts both within red hat, but also really promoting it out in the community. This is everything from website that we we built, but also developer, advocacy.

E

Well, nice to meet you Andy.

C

So uh I had been given a uh direction that we needed to be able to generate reports uh on deployments and uh products about uh open cves uh on s-bombs uh salsa information. Ssml is like our Red Hats version of salsa and adherence to a variety of compliance standards and I realized. If we had that type of information or reports, it would also help in aiding CDE remediation and support policy enforcement.

C

So uh I'm gonna do just a quick review of of that's kind of how we got. That was the the impetus for how we got here so uh we're gonna do a quick review of and chorus again. So it's uh think think Emporium when you're saying it uh not like empress and it's Greek for merchants or Traders, um and let's see so, we we can use emporis uh backed by any type of uh oci registry.

C

That's uh V, 1.1, compliant uh to store a variety of information and but that type of uh radical change, maybe not everybody's ready for uh immediately so trying to meet content and people's appetite for change. At the moment, uh we could store CDE information, trustee data and containers in the oci repository, but we could link to using a package URL to other types of artifacts, for our trustee information to to you know, bring those two together, so I I want to do.

C

You know somewhat of a demo, for you guys today um we're gonna go use case and then demo, and maybe some follow-up use cases for the same demo and then repeat that again. So the the first use case here would be for software. Developers adds us just so. I want to set policies to prevent teams in my organization organization, from using untrustworthy libraries and packages, and imports can enforce these uh or it can provide data to enforce these policies and prevent a user from installing something that they probably shouldn't install.

C

uh So this this would be the case for Pips uh uh Maven uh libraries, yum Etc, uh and we're going to do a demo now of uh a pip plug-in that will that illustrates this. So, instead of doing a I'm gonna do a recording.

C

No, it doesn't want to go for a bit all right. So, let's see.

C

So, uh first we're going to start with an open oci registry and we're going to uh pushed a bunch of of Pips up into the oci Repository.

C

uh I guess actually, first birth, knowing there's nothing up our sleeves um and we're gonna start running the the registry and uh now.

A

C

We're gonna push things up there.

C

So those are the Pips.

C

And we use uh yaml uh config similar to what you would see in kubernetes to to configure and and control what gets passed in.

B

Now is that something that we would be using on the ortelia's side to push up metadata that we want into yeah.

C

This is actually kind of a a low level looking at the plumbing uh part of this, so uh different clients could integrate to use the the API that we have to. You know push their their uh metadata up as they as they like.

C

You can see that's not in our path, we're gonna, try to install it.

C

And it's not where we would normally find it up in Pipi.

C

Now we're going to use our Plugin or wrapper.

C

Now we have a virtual environment for our tips to be installed into and we'll install from our pip from emporis or through imports from the oci registry.

C

We're going to use it there we go, you can see, it's actually working.

C

And he's using it right there for displaying a Beat.

A

C

We also have a fuse driver that allows you to mount information. That's uh that's in emporis, so that you can look at it uh as a file system as well.

C

But you can see the metadata coming up there through uh the file system, attributes.

C

That part makes sense where we're we're not creating a pip uh index and storing it in in Porous that that is uh generated on the on the fly. So you can add add new Pips and you don't have to or if you were, you know using it for an RPM, a young repo. You wouldn't have to do a rebuild the the RPM index. So all this type of stuff can be generated on the Fly.

B

So how would a developer as um this creating a python module upload it to the registry where they still do like a twine uh upload or they do? You have a wrapper for that? They would run.

C

F

C

F

Speak to go ahead.

A

F

uh You can use twine if there is a translation layer uh right now. The translation layer is uh the shell script just shown at the beginning. That builds the uh the repository from the wheel files, but the yeah, the the point. Translation layer uh doesn't exist yet, but it does use the uh regular kind of like simple python index uh uh apis. So it's probably not that much extra effort to implement.

B

Right: okay and then um at what point, is the s-bomb uh generated for those um modules that are in the registry? Is it when they're loaded or um uh on the Fly.

F

In this specific demo, uh that's not implemented uh we've kind of got separate, piecemeal demos that we intend to combined together in a more Unified package, but the s-bomb stuff, when uh combined with this, would be generated on the fly as uh as requested and the information used to generate it can be populated at at any time, and uh the generated spawn stuff would just be uh used from the data existing in Porous.

F

C

Hold On Let's: uh let's choose some different words, maybe um so the the generation versus aggregation. So we we uh for the the PIP when you're publishing the PIP. That's that's when the F-bomb should be uh should be uh the s-bomb information. Probably mini s-bombs is our approach that is in small, would be published and then, when you want an s-bomb, uh uh you know you, you you're, you can request it in whatever format you want.

C

So if you say, I wanted an spdx, um we'll look at what you requested for and we'll go grab all the mini, s-bombs and aggregate them together and translate it into the format that you requested. Okay,.

B

All right, so it's kind of like a two-step, well, not really a two-step, but it's.

C

On the publisher yeah, when you're publishing the PIP uh you, you should have the the s-bomb information, then because that's we store it together and it's signed. So we don't we don't if you, if you stored something without s-bomb, it gets signed, and you can't you can't add s-bomb stuff later, because it's already it's already signed and has a um uh a digest. So that would change the digest.

C

If you add uh other information and it's immutable, so you would have to publish a whole new, a whole new uh version and and then you could, you know, marry some some s-bomb information with your your artifact.

B

C

B

So how does this differ than well on the signature side, um Sig stores, doing a bunch of signing so like for a ruby gems package, they're going through and signing the the gem with Sig store? How does that work with this.

C

Well, six stores its own thing and the ruby gems are a separate thing and then you have to you have to link them right right. Okay, here they're, it's all stored in the it's either all sort in the same place or there's a p or L back to to where the uh the Ruby gym is. So we we offer a path forward to eventually storing everything uh together as well. uh We we support uh uh user defined schemas for additional metadata to be published at the time of generating a pip.

C

If they're a you know, they have their internal software department and they want to add extra metadata in their pipeline. They can do that and then it would be signed with the the artifact that they're publishing or if they're, if, if the artifacts already exists, it's got its s-bomb information and they have more metadata that they want to link to it. We can use a refers API to uh to link to the artifact and uh their their metadata can itself just be signed.

C

So there's more flexibility here than just with with six store.

B

Got it, but you would be you, would you consider yourself a competitor to Six Tour, then.

C

Yes, okay, Luke Luke Heinz uh uh is is uh uh working with us on Imports as well. Foreign yeah.

B

Yeah I know uh him and um I can't think of his name. Now, let's start at six store and chain guard anyways.

C

C

B

And does that does that API exist right now to be able to add additional schema metadata to um the items.

C

um Jen, do we I, don't think we have that exactly yet? Do we.

G

I I, don't think I fully digested the question. Can you please repeat it so.

B

If we wanted to use um emporis with our own metadata- and we want to associate additional um schema metadata to an s-bomb- um is that API available.

C

He Jim he's asking: can he register a new schema.

G

Oh yeah yeah that can be done, but right now.

G

And it's a prototype, we don't have like complex uh metadata yet, but we're adding that so it's really only Primitives you can have so that might not be helpful at this point because I'm sure you guys have like complex structures in your metadata, but uh that should be added pretty soon.

B

Yeah most of our our metadata is going to be like a service owner and the service owner is going to have an email, a name, a phone number um which are all strings. um So it's not like a really overly complicated. um You know schema that we would be adding.

G

Right I think our end goal is to just be really basic and support like any Json schema um since manifest or short and Json. Anyway. It makes it pretty helpful.

B

A

C

Gonna turn constraints I'm gonna skip to next uh use case that is, has is paired with a demo. So, as a user given I have a CDE I want to see all uh contents that I own, that is affected by the CDE, so that I can put together an impact analysis report and start uh remediation plans.

C

A

C

Don't know how are you, okay, sorry, uh I can go faster this time, so demo objectives, uh we're gonna, discover existing application within a course. um Then we're going to run that guy we're gonna, publish some uh updates and uh also publish a CDE and then discover those cves and discover what uh applications are linked to that CDE by way of a a granular artifact.

C

So we'll start our demo environment get that up and going and we're going to discover existing application.

C

So we're again using uh yaml like you, would be familiar with uh with Cube, cuddle or or uh OC.

C

We found our our uh container and now we're going to run it, so this is uh uh plug-in for containerd. That uh now can talk to import us, and so we have the the that application up and running.

C

Now we're going to publish a new version because we found a bug.

C

So it's it's remember. This is a lot of the the plumbing behind the scenes that we're showing how this would work, uh but a a better ux would need to be uh put on this and it would be you know, specific to like podman or or whatnot.

C

So this is remember again. This is low level Plumbing that we're showing here, but it it is uh the same old, familiar uh build, publish, run or build push uh workflow.

C

And we had one application actually dependent on another one was hello world and this one's hayworld and we're going to make them dependent on each other and they were dependent on each other, and now that we we updated hello, world hayworld needs to be updated because of that dependency.

C

Now, after doing that, we are going to publish a new cve.

C

That's the cve.

C

So it'll be build and push.

C

A little silly, uh but we're going to now show the the cve information. We know what the the cve ID is we're going to search now for that, so we can get more information and what it links to.

C

Now we can see it links to two different uh applications, hello world and uh therefore also.

C

C

We're actually sorry, it was two hello worlds and two different versions of fellow world and.

C

So does that make sense how we were able to to publish the CDE and it links to um uh already published artifacts in the OCR repository so.

E

I got a little lost. How did you link hey world and hello world.

C

Well, let's see so if we go back to the the CV e itself, so.

B

It was before before the cve was created. You linked the two together yeah.

E

And I just didn't know how you determined that they should be linked.

C

So the CDE effects uh this version of hello world and this version of hello world. So when the cve is published, you have to say what granular artifact that it that it affects. So this is this is inside of hello world. This is an artifact inside of hello world, so so.

E

But but you said you had you linked hayworld and hello, world and I was just curious. How you knew that those two were linked.

A

G

On that instance, is is really like uh we're saying: that's like a developer decision like I as the hey World developer, knew I had a dependency, it could even be like say: I uh was building a terraform uh module and I needed. Another module and I and I knew I was gonna have to pull that down. Then I could link it and create that relationship.

E

So, okay, so it cut us the same way. We do it. We have a give us a package. This is what are they can? What are the microservices you're dependent upon? So if one of them have a cve, we can report up to the The Logical application.

C

Yes see it, it's a clear. So instead of uh you can do a query that says: uh there's index manifests and then there's artifact manifest, so the index manifest links to artifacts manifest, and this cve links to artifact manifest. So you want to know the index manifests that are linked to artifact, manifests that have that cve also linked to them. So there's a there's, a new search API uh that we we would. We want added to the OCR spec to allow us to do those types of of searches, foreign.

E

The type of search that says, uh uh if I have two, if I'm hello, world and I'm dependent upon hayworld, that, if hay world will tell me if I, have an issue.

C

Well, uh first, it would be uh show me: the uh index manifests that linked to an artifact manifest. That is also linked to this specific uh cve.

E

A

C

E

Can Traverse the other direction right.

C

Yes and then you can, you can expand it from there so like what other, what other uh index manifests linked to this index, manifest the one that we just found, so you can keep keep going up, so you can say ah so. The index manifests at that level would be like a component and then that component could be used in several different uh products and that product could be used in a platform or different. You know things of that nature, so you can.

C

You can Traverse that uh uh uh dag to find all the the affected index manifest in the repo.

E

So do an architectural diagram that you could share with us.

C

I, don't know that this qualifies as architectural.

E

Yeah I, don't think you did I, don't think this one I looked through it and this one didn't have really kind of a complete kind of architectural diagram. So.

C

If we have the cve and we search for the artifacts that are affected by the cve and then the uh index manifest here in RPM and I'm an image also an image, we can that's how we would find that say this image or this RPM are affected by this CDE. Then you could take it a step further and say this being more generic.

C

Now this component, or maybe that's a image or an RPM Etc- are uh also affected by an artifact and therefore they they also roll up into this product, which is also an index manifest. So you can keep that that chain of index manifests uh you can you can search for all of those? As a uh in your query,.

A

So I would be looking for a logical architecture to start with. That says here are the chunks of Taurus and here's how the flow occurs amongst them. This shows how it looks at the product, which I think is a little different thing. In fact, quite different.

C

A

C

How it looks in the this is how it looks in the oci Repository.

A

C

So then, then it becomes about the the uh search endpoint to be able to to link those those things together and I.

B

Think that's what Tony is talking about is the the architectural design around the search endpoints.

A

ah Yeah, so there would be a data store which at the moment, I don't see in this diagram and well you know control flows. That sort of thing.

C

um Let's see so the oci spec and how it's implemented so we're just proposing a new endpoint that offer. You know that you can uh query and, and these types of ways how that is implemented behind the scenes is, is up to each oci repository implementation.

C

So is the diagram that you're looking for actually part of the oci. However, you would implement it. However, you know Quay implements it or or uh that or Docker Hub. Those aren't those uh the architectural diagram that you're asking for. Isn't that a an oci implementation detail or am I a misunderstanding?

C

A

Probably is to some degree I'm looking at how and Taurus sits on top of it.

C

So enforce is going to uh use that that uh search engine or that that search endpoint excuse me to um uh provide the information that is being asked by whoever's, calling them for us when they're doing when they're they're asking for show me this or show me that, okay, so how how it stored in the in the background, you know that it implemented it just we. We just need it to look like this uh when we're asking for it. How would that how it's actually stored in the background?

C

I, don't know uh uh that we'll we'll see how that you know that's an implementation D is held for for those oci registrations.

B

So with what we're looking at here and the um this aggregation that we're looking at is um the result of a hitting a search endpoint. Is that correct.

C

No, this is how we publish it. So this is. This is uh how you you would, uh uh when you're publishing the product, uh that's a an index manifest you would you would link it to the components that make up the product, so these these links would be done by Digest.

D

B

Right but let's say I wanted to look at that that product I would go and search the oci registry for that product and I would expect. As a result, I would get a Json of the product, the extra all the components, plus all the components extras, plus all the does that exist.

C

B

That endpoint to search a product and return this picture.

C

We have a uh when we were doing our demo there. We have a modified version of the uh go uh oci registry- um oh Alex, isn't here uh Jen, uh that's that's the the cncf, uh so registry. Is that correct.

G

That's correct: um we would still need to do some sort of um oci extension sort of um do a write-out of that, and that might be what you're looking for that. Just to tell you what the expected inputs and outputs um so yeah, that's that's um not finished yet.

B

Okay, I I think that's what we're looking for. So we had we're trying to get a better understanding of how we're going to interface with this and use it. So that would be a big help. Can you go back one screen or one slide.

B

um On this slide, you have the container on the bottom left.

B

uh The bottom container is going to contain RPMs that were installed into that that image.

B

um How is that represented.

C

um Right now, the way that this is showing is this is just a bunch of files, as opposed to uh you know, you can have it, it would, if I, let's, if I meet so.

B

Let me let me re-ask the question, so would it go from container to RPM to RPM to artifact yeah here.

C

We go, it would look like that. Okay,.

B

Okay, just want to make sure we're on the the same page on how these dependencies are are traversed.

C

Yeah, the the reason or the the point of showing it of showing this is. We there's no relationship which, like the container, doesn't include this RPM, but the container does include article share artifacts with the RPM shares and like they both contain those common artifacts. So when you're building your your s-bomb for a a consumer, uh the the s-bomb for the container and the s-bomb for the RPM would be generated by all the mini uh s-bombs here. So there was when it's stored in the oci repository.

C

These artifacts are only stored uh once, but they are linked to by both. You know they can be linked to by multiple index, manifests.

B

Right but if I went into the container and did a yum List It would show up that I have RPM dependencies. That's where I got confused.

C

Yeah, if we, if I put it back to this, yes, that would show that you have RPM RPMs installed: okay, yep.

B

um Just to jump I don't know how much time we have left here, uh just to jump topics a little bit. um What is I take it to to make all this. um The oci registry run is going to be pretty much a standard oci installation is. Is that the like you're saying the cncf go registry um is one to look at and how it would be implemented at a customer site.

C

uh We well, let's see so we modified and a a uh the the cncf go registry, so we have a fork of it um and I think the way it's done was mostly for experience sake for us to produce demos, showing you know like a minimum plausible uh type thing. So it's it's. The way we did it I wouldn't say is uh it was for expedience.

B

Okay, but if we wanted to to understand so, let's say um or to use oci registry, as for lack of better word, it's it's database um in order to get understand the implementation needs we'd. Look at how an oci registry is being implemented at inside of a company. Is that a correct assumption.

F

Even in the side of a company.

C

I understood every word, but I don't know what you need.

B

Okay, so I am um a MasterCard and I want to have um my internal software be published to and track through emporis in ortilius.

B

What would I need to do to track my internal software and track the open source software that I'm consuming? uh From being you know, internal to my company as MasterCard.

C

um uh So ortilius right now, you guys have documents for like this is how these are the things that you would need to do in your pipeline. Mr Mrs customer um to push things through ortelius for us to be able to to then do all the the nice things that we do later on for you to manage your your uh your deployments and your applications. So right now your your microservices talk to a uh uh uh postgres database, correct, correct.

B

And that postgres database would be internal to the customers um environment there and we would be pulling things from uh an external postgres environment as well uh to aggregate together.

C

B

Now, if we're going to replace the the postgres piece with the oci registry, how how's that handled in in the aggregation so.

C

Let's say they already have a uh uh a a.

C

Oci repository that would implementation that will work with uh Imports. So it's up to the right version, whatever like that, um then I uh in Porsche would need to be deployed. You know, maybe it came with that um oci repository um or or didn't, but your microservices I would suggest that you, you change your micro services to points to uh Imports um or um or you know. That would be a first step and then you guys can look at later on. Do you wanna?

C

Do you want to just use the emporis apis instead of the the microservices that you have have created.

B

Right right, yeah exactly but on the um okay yeah. That makes sense.

B

And is emporis able to aggregate across multiple uh oci Registries, then.

C

Yes, well uh so.

A

B

If I have one that's being run by the C, uh the CDF, that is uh oci registry of all of the um yes.

D

Resource packages yeah, you have a way to reference content in various oci Registries I'll open one.

B

Okay and that's just based on the the links in the schema, correct, correct.

B

And now I'm going to put you on the spot. When are you playing into releases.

A

C

We uh we, we don't have a plan to release that, yet we're we're at the point of like should should will this. uh Do we have enough people that would help us uh build this? You know we're we're.

D

Doing a lot of efforts in the community so we're obviously Red Hat, being Community First working to build out what the community is going to look like around this and then based upon that. Hopefully you know we'll be able to you know, get more contributions and add in all the features and put a timeline together, but yeah we're working on the community enablement as well. Okay,.

B

And then on the um uh has it been decided whether you're going to uh request the oci spec changes, or is it just going to be um extending the existing oci spec? Has that been figured out yet?

B

Does that make sense on my question.

D

Right well: go ahead. Jay.

C

uh Oh I was gonna, I was just gonna, try and pass the the mic to um to gen Jen how you you were wanting to use that.

C

Proposal Doc: it's it's up here, right.

G

Yeah the um what I would say up there is not fully vetted in fact this week we're looking to kind of uh fully vet our approach, but I. Think at this point we would like to possibly make changes to the oci image spec pretty generic changes, and then our approach is to do a distribution, spec extension.

B

Okay, so it's gonna be a little bit of both.

G

Pretty much okay.

C

Jen does did I interpret what you just said as we wanted. We want to do a little more work privately and then we'll share it with them.

G

I would say yes, because what they see right now wouldn't be uh is, is most likely not going to be our Final Approach I would say.

A

C

So as soon as it's not confusing, we'll share it with you guys, okay.

B

C

um So we got like 12 minutes, left, I. Think.

E

C

E

That was it yeah I. Think the next steps I'd like to do another follow-up call.

E

um So we could have time to uh kind of digest what we've learned today and I I would very much like to have more of a um if this is something that we think that we could do some proper integration with and it, and it makes sense to have more of a discussion about what what benefits we're, bringing to the table, what benefits uh you're bringing to the table and what is a uh you know, what does it go to market strategy actually look like um as the two of us working together to solve, because we have very aligned goals, I think we're taking it on somewhat differently, but our, but the all the end goals are are very aligned um and there are some things that we do very much similar.

E

So I think that that would that's probably what we should do next is make sure that we don't have any questions for each other and and have that discussion of what an integration would look like and who are we bringing that integration to and Andy? Did you get a chance to see the recorded version? Yes,.

D

I did I got a chance to pass out yesterday, so I'm very familiar with the prior conversations and I'm. Definitely looking forward to continuing with conversations that we've been able to have thus far now.

F

D

Future, because right now, I'm I'm focused mainly on the community building, so anything we can do to help promote the community if things are unclear.

D

I'm your good point of contact to help build that repertoire out and build that content. Okay,.

E

Maybe schedule a call for next Tuesday. At the same time,.

C

Sounds good okay,.

E

And Jay have you updated events at all.

C

uh Yes, I did last Thursday.

E

Okay, um is there any point in time that you think it would be beneficial for him to be on that call.

C

um I don't know, let me think about that. Okay,.

E

All right, Daniel.

C

I, just don't want to leave.

E

Him out of the loop I know these low-level calls don't make sense, but I don't want to leave them out of a loop that he may want to be involved in.

C

I I um Daniel Messer joined the call also um he he uh leads up uh Quay The Quay project, um so I I think he he would foreign be good to have involved as well from uh having an oci uh registry that that can uh lead by implementing these uh proposed changes before they're necessarily accepted by the oci steering committee.

E

And Daniel: did you get a chance to look at to see the video or do? Is there any? uh Is there any clarifications that you need from the ortilia side.

F

I was just halfway through when this call started so I plan to finish the video later.

E

Okay, well Andrew or Daniel. If you want us to um get on a 30-minute, quick call to catch you up, we're happy to do that. I always find it's easier to get everybody on the same page as we move forward than to try to get somebody caught up later.

B

And I I have one more technical question: um is it possible to have in the oci registry.

B

um Just the metadata without an artifact, you.

A

B

Be yes: okay, because I'm just thinking the path to adoption, um there will be cases where things would still end up in Pi Pi, but not in uh the oci registry, for example, for some customers.

B

And I wasn't sure if oci could handle that. Yes.

C

So that that's one of the changes to the Manifest uh that we're looking for so the the artifact manifest instead of having, uh instead of having a blob used in it to store, say a jar file right, um we would and instead use a package URL to point to where that artifact is located. Okay,.

B

Got it- and you said, that's one of the the changes to the spec that you would you're proposing. Yes,.

C

Because, like like, you were saying like so, the this symbol appears like Ah that's way off in the future. You know, hopefully we can get people to migrate there, but that's probably a bridge too far at this point. So this is how we could meet people now.

B

E

Well: okay, everybody. Thank you uh so much for um spending an hour with us and talking about this really geeky stuff.

E

Away this last hour, you tell about cves and s-bombs and aggregating data, but I know it's going to be important for companies as they move forward, especially as they really build out microservices. That's when that really they start really struggling, so I sent a invite out. uh You should get it everybody on the current. Invite would have gotten it um for next Tuesday uh from at 10 o'clock. Just so that, just so you know we do have our Community Toc meeting at nine. So if we jump on a little late, we apologize ahead of time.

A

No problem, but.

E

We usually keep it try to keep it to an hour.

B

Okay, thank you so much for the Deep dive and.

E

I'll, send everybody on this list the the recording uh the link to the recording as well.

A

C

E

B

You thanks everybody.

E