Cloud Native Computing Foundation Security Special Interest Group, 26 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CNCF SIG Security 2021-04-26

Description

CNCF SIG Security 2021-04-26

A

Hey matthew cameron, can you hear me yeah, hey how you doing? How are you going great great, really? Well, it's been a while yeah. It has mate well over here. Yeah absolutely. Did you get down to b sides? I did not. Unfortunately, I had to stay in sydney, but uh how about yourself? No.

B

Mate not the same here. It's a bit crazy with me now with two children, so oh wow, great.

A

So when was the second one born uh nine weeks ago, mate, oh wow, fantastic, that's great! So what's what's their name, yeah a river river, oh sweet, yeah, yeah, sometimes yeah, absolutely yeah, it's good so, uh and the other one's max right, yeah! That's right! Yeah! That's right! Yeah! I remember that oh cool, so he he it'd only be like under two max is 18 months.

A

Two under two minutes yeah you wanted to keep them together. I had I had three under four at one stage, so it's the way to do it, knock them all out, keep them all together. They become great mates, yeah.

C

That's what I'm thinking! That's what I'm.

A

Thinking yeah and uh how are things going with so? Oh good, good yeah we've been um smashing it actually, so a bunch bunch of really good deals and um doing a bunch of stuff with with uh with actually we've actually had a whole bunch of new products recently.

A

So we bought a company called musedem all right, which is basically a sort of a uh static analysis product a bit similar to sony cube. I guess yeah and just to expand our coverage of that space, because a lot of people want that as well. Yeah sort of single throat to choke yeah, no for sure yeah, and then we.

D

A

Got a strategic partnership with uh neue, vector sort of adding the the containers, the container piece as well? Okay, what are you guys doing.

B

With containers with new vector yeah.

A

So, basically, just working with them, because they've got this really clever, behavioral analysis stuff to do dynamic, sort of port blocking and process blocking them and sort of teach it. You know programmatically what processes are good and bad and that's sort of pretty innovative. No one else has got that so yeah that that's sort of the differentiator, so you don't have to talk to the city. You don't have to sit there and whitelist all the processes. It'll actually do it automatically based on you know, putting into a learning mode.

B

Yeah yeah for sure all right that makes sense. Hey everybody else, gonna.

C

E

Hey, how are you.

C

Hey adam, I'm good mate, I'm good! Are you catching up with me this afternoon, mate.

E

C

B

E

Oh yeah, we can do yeah, yeah yeah. I forgot about it if we were but yeah. No definitely.

B

Six o'clock.

E

ah Yeah yeah, I remember that yep yep, that's fine!.

F

All right, nice, nice, nice, um might add in my marketing person as well to be a flight on the wall.

F

um Yeah cool man.

B

All right, how's, everyone doing.

G

Terrific just nervous as if before every presentation matt, you know.

C

All right well look at 12 o'clock. We might as well get the ball rolling. um So thanks. Everyone who's here, um hey justin,.

C

Might hand over to our fearless leader, justin hey he can handle it. Hello.

C

All right well, he's he's muted.

F

But I'll just finish what I'm gonna say um thanks everyone who's here so far um nice to see you all again for those who, I already know um I've organized with with andreas here from red hat, to give a bit of a presentation on software supply chain, security.

C

By the usage of software factories, if you're familiar with, um if you're familiar with the dod devsecops.

F

A framework um as part of platform one they they introduce this concept of software factories and it's it's pretty cool, so um he will handle that. um I thought I'd copy a note from the the previous agenda on april 21, just to remind everybody that well, kubecon is coming up and as part of that there's cloud native security day on may the 4th. um So if you haven't already check that out, I think you you're going to go to it. Aren't you brad.

E

What's that sorry, so could.

C

You even go to kubecon.

D

E

Do yeah definitely yeah yeah. I normally go each year, my um it's not good timing, but um I just forced myself to stay out.

C

No, it's not. Is anybody else going to that just out of.

F

Curiosity, I'm going to be there in spirit, digitally.

E

We should do a little um zoom meeting to hang out in between or you know, yeah we.

F

Could yeah we could have coffee for the first couple of hours and then beer.

E

I normally have an irish coffee, so I put um whiskey in my coffee or baby yeah.

C

F

All right well, look. um That's kind of all. I wanted to say um justin, I noticed you're here. Do you want to do an attack over.

H

um I think it might be product. My system is saying it's going to do a software update in a moment, and I have a feeling that that means that I'm going to be disconnected so um uh you've been doing a great job. uh Let's, let's have you just.

F

Cool man all right, we'll look on that note. um I hope your system goes. Okay, do you want to just get started andreas with this presentation? I think that's why everyone's here and we're all pretty excited to to see this.

G

Sure mark do you want to share your screen and um and I'll also say hi to um we've got uh other red hatters on the on the call today, as well.

G

Andy block from um from the us he's one of our principal architects there and um our distinguished architects and we've got adam goose and shane bolden on there and mark hilton brand, and we all work for red hat, but uh we all sort of also you know, are interested in this topic and and I'm probably the one who's done, the the least amount of work in there so cutest doesn't belong to me.

G

It's more the people I just mentioned right, especially mark who was building this um this demo in the last few weeks and um so yeah. Let's kick off so um where did the idea come from? So like many many a few months ago now, right um many weeks, I basically watched a recording from one of the cnc f6 security uh sessions in the us, and then john meadows was heading. This up and and andy martin was sort of presenting the the concept of a software factory, and I posted some screenshots in there right.

G

So multiple csd pipelines can be composed into a complex build system, and this is called a software factory and it's basically, you know, used to securely build and deploy all components of a system and at the same time we were then writing that white paper about supply chain security, and we learned about the solarwinds attack right and then we looked at this. This architecture diagram on the on the right hand, side and and mark- and I basically looked at that and and go like what are the components and how could we make this better?

G

Then we reached out in in good old, open source fashion to the the wider redhead community across the globe, and we actually found that that our teams that work with federal governments across the globe uh specifically have already started an open source project called ploygos and we've got the we've got the links in here, but that that is actually the you know open source way of um or approach to basically build such a software factory.

G

The only thing we didn't quite I didn't quite like about um the software factory secure bootstrap- is that it sort of the idea was to have a laptop locked away in a world where you basically would start off that, and- and I thought in a modern enterprise that probably doesn't work that well, so we need to come up with with other things and even though boot attestation is not part of this demo today, you know, I think it has a lot of vital building blocks to actually you know, bring those secure software supply chains into into enterprises, and then you know, as you can see, what's the problem- um that's not from us.

G

That's from the previous session, like large problem space requiring an end to end solution, and that's really what this progress is um is all trying to do and on the other, on the left-hand side, we've got the the sort of famous in in the six security space right, the dod enterprise devsecops reference design, and so I think, all the teams that was sort of a common denominator.

G

We all knew about it and- and we all sort of you know, agreed with that approach and- and that's also why what you see today is aligned with that uh reference design so mark. Do you wanna get to the next page, so in good again in google dot open source fashion, the ui is probably the last thing that gets updated.

G

So what he sees is a flurry of text messages, I'm just going to run you through a couple of screens so that you get used to that when you see it in the demo of what it is. This is really just a text output and this usually pass messages in there, but the the orange highlighted text is basically fail. So that's why you know something is is wrong. It was interesting.

G

Mark told me the story because we we were like mark was rebuilding that many times over right and and it fails often because you know some some parts of the of the project.

G

The open source project have been updated and then you need to sort of re-check all the parts and then, when it went this time you know mark was thinking ah now what changed now and then he realized actually nothing changed, but the software supply chain did actually its job, because what it realized is the, as you can see, the title for this step is ensure software patch is installed and there was a new vulnerability and by checking the content, it actually checked and and realized that the patches have not been all updated and that's why it failed- and that was a great case um to to show us that this is really a good approach and it's working all right.

G

So next slide, then this is a. This is an open shift screen. This is the deployment topology, like all those components are being installed by an operator and that operator basically takes care of all the components you need to run and set up your your software supply chain or your software factory as such right and we go through the through the components more in detail during the demo.

G

As you can see, there's gt on there there's, you know all those components in there that actually makes that a comprehensive solution right and if you think of enterprise context that that's actually what you want. You want the single operator certified operator installed, and then you know that everything is taken care of, and you can trust that and and sort of you know, build your enterprise software based on that. So next screen pipeline view.

G

This is um this is just our jenkins pipeline and um the blue ocean view and what what it shows you is exactly the error message I said earlier in the middle, you see the ci static image scan is failing, and, let's just uh you know, show you how it how it looks like later when you encounter that screen. So the next screen and then context so that the ploy goes project is, is not a product right. What we? What we uh do?

G

We want to invite everyone who hears about this to contribute and make it a successful, open source project. It's um it's! It's a redhead uh led project at the moment because our consultants, all across the globe, wanna, wanted to collaborate, and you know that's how it got created. So it's the the perfect storm, basically for uh for an open source project, um driving faster results for our customers without reinventing the wheel and um yeah ready consulting. I mentioned that the dod website to align with the white paper and reference architecture.

G

I mentioned that and then um yeah. The security meeting that I mentioned earlier was basically the starting point to for me to to think about it and and gather around and then at the moment it's teams in australia in us um that talk and um communicate and collaborate across that- and I mentioned the laptop in the fault that we didn't didn't like, and um we also have everything you see today is um is in the script and that's uh on uh on mark's sort of github account and the main component.

G

I mean there are more components on there, as you just see. So, like the operator, the the um you know, progress project and there is the the gt um source code management system in there as well, but just as a on a high level right. The openshift platform is the user interface that you see on the right-hand side. So it shows you all the the operators that we have here installed and then uh yeah program.

G

Software factory operator is obviously the the main one in this case, and then reusing or mark has used record for artifact at the station, and I think uh that's it uh next slide is that where we move over into the demo mark yeah, so.

I

You move over to me: yes thanks andreas, so yes, I'm a mark standing on the shoulders of giants. So I didn't deplo, I didn't create most employer ghosts, but I have used it. uh You know I'm a longtime listener, not often that I call in, but today I'm going to show you a demo of how plugins works I do have.

I

I do have a live cluster which, with all this stuff going, but just because builds, take a long time and demo gods are binge full. I do have it pre-recorded, so we can get through all this in a reasonable amount of time. People can still have lunch, so the demo there's a lot. There's like a couple: different demos, seven different little snippets I'll show you uh but broadly categorized into three chapters.

I

If you will so the first one is sort of riffing on what andreas was just talking about like okay, so we have the department of defense they're talking about things like software factories, employee ghosts. How does it make a software factory? So we've sort of hinted at that again with this kind of this? Is the developer perspective of open shift showing you a myriad of boxes, which is probably slightly confusing, we'll try and make more sense of that in a second?

I

But what you'll see in this demo that I'm about to show you is that you start with the flagos operator and we'll talk about? How does that even get in your system that can also be available for air gap systems with which adam goosens who's on this call?

I

Who he's done some contributions to plegos for even being able to support agencies, government agencies that can be connected to the internet so a little bit closer to a laptop in a vault which andreas and I like to joke about, but anyway, so the plugins operator is in some ways a route of trust.

I

If you will, the idea is that when you create you'll, see me in the demo create a platform custom resource in true kubernetes fashion, and the operator will read that and from that it figures out what factory, if you will to create, so we're not building anything yet we're just building the things that are going to be building things or verifying the things we're building.

I

That's where the operator will busily work and you'll see in times like time lapse fashion. It kind of builds all this stuff out. Just because I created one little custom resource. This is all the magic of kubernetes. It builds these all out, and the openshift developer perspective gives me a view that demos really well so I'll show you what that looks like again. If there are questions, I have a live cluster and all that, so we can get into that, but this is a little helpful. So we'll stop. It start with the operator hub.

I

The important thing here is: you see this notion of the provider type so that provider type says an operator hub is basically saying: hey I'd like my cluster to connect to you know all these other operators that are available in a marketplace and some of them are community operators. Some of them are certified by red hat, because this is red, hat open shift. What you're looking at right now, in our case, the plugins operator, came from a separate kind of provider which I've installed on the cluster previously. So as an admin, I've decided.

I

Yes, I want what they're selling I want those operators right. This is what our community, as andrea said. Our consulting group put these operators together, and this is the ploy goes sulfur factory operator, which I had pre-installed as I, as I promised you on this cluster.

I

If we look at the operator in a little more depth, it looks like any operator you see, the install of the operator succeeded and it looks for two custom resources, a pipeline which we'll get to in a platform which we're going to look at now, there's a form view for the platform sort of I'm going to use the yaml view we'll come back to this at the end. So this is basically prescribing deploy, goes what we want it to build right, that's the operator. That's in the developer perspective.

I

That operator then just starts building things through its blueprint. It knows what it needs to pull in now, it's pulling in things that go a little bit beyond just the pipeline, get t you see pop in there, it's a little bit chaotic selenium is coming in there. It's like a thousand little cobblers, elves running off and creating a sulfur factory for me.

I

So this takes. uh I think it took about sevenish minutes uh when, when all was said and done, if we take a look at it, this is a platform that is revolving around jenkins. There is also a tecton version because who wouldn't want a tekton version, I'm showing the jenkins version. That's what I started on and you can see like. I said there are elements in here that aren't related just to building there's also a ide called code ready workspaces based on eclipse j.

I

If we look at the platform in a little more detail, so now I'll just compare and contrast the custom resource with what was created so getty, you can see get to use over there right. So basically, the things that I called out that I wanted for my continuous integration. The things that I wanted for uit jenkins is there that'll be important, static analysis. Sonar cube is in there good good for good on us, for including solar cube nexus is in there.

I

So there's a number of things. It's basically. If you will an implementation or um you know, the the department of events. White paper is prescriptive in terms of what you should do, but not what tools you should use, and these this is sort of our consulting team, saying hey within red hat. These are the tools that we find are the best practice for implementing some of those different stages that every software supply chain should have per the white paper and per our collective wisdom as as a community.

I

So that's that's the platform, any questions about that. So far I realize where andreas and I are kind of dominating any anything anyone wants to add or any questions anything that's like totally doesn't make sense. I had a.

E

Question, can you just go back to the young file to just the previous slide, so we're saying here that you can choose these options and if you want to manage you know you can get argo cd, as that essentially runs a helm chart in the background or something and does all your persistent volumes and everything.

I

That's that's right, though there is a. There is some proviso in there like as you'll see when we get to the end of the demo as long as certain things you've implemented certain things ahead of time, but yes, it's been set up so that you can plug and play which you see is best. So it stands to reason, if not argo you could put in what's called a step runner you could put in something that allows you to use flux or something like this.

I

uh If you wanted, in our case our platform again just to give you in the live cluster, what this looks like, maybe a little bit easier to see. This is the yeah. These are the different options that this platform.

I

This is what we tend to use in red hat land getty for an internal get repo jenkins, selenium yeah. Does that does that make sense, yeah, yeah, definitely yeah. It looks awesome, cool cool.

C

um Yeah, just is this specific to openshift.

I

uh It's not well, this is the thing, so it's not meant necessarily to be specific to openshift, but for our red hat consulting. They tend to use openshift because they're, red hat consultants, but most of the primitives we're using are generic kubernetes. If somebody else in the call wants to jump in on that yeah.

J

I

Know I guess I can add something marked.

J

um So yeah, so a lot of the primitives, as you mentioned, are using the with out of the box with kubernetes. The operator pattern is one that isn't designed for openshift, it's one that can be deployed in any kubernetes environment, because it's just running a control, loop, custom, resource definitions that are applicable to any environment. It just comes down to what you want to enable and how.

E

And um does it um account for let's say you don't have enough cpu around in the cluster to to install them? Does it just blow up or it sort of nicely tells you you've.

J

So right now it is kind of um prescriptive, so we have a prescriptive stack but down the road we're going to be looking to seeing different ways to provide more capabilities.

J

As we mentioned on the c on the ci perspective, we have jenkins or tecton uh to be exposing more options down the road because we especially because I'm field facing so I see a lot of customers and some want to use one product- some want to use a different product, so we're going to provide better options down the road and provide ways to enable and disable certain features as necessary.

I

And as someone who's experienced a fair amount of failure at the hands of this platform, one way you find out is in typical kubernetes ways. Like you see here, the operator has installed successfully. You could imagine a world where, if I didn't have enough compute to create everything that the polygons operator needed like say, code, ready, workspaces or eclipse changing this would be. It would be in a reconciled. Loop.

I

What's that sorry, I didn't hear that.

I

Okay, cool but.

G

It's just it's not just like what what's redhead providing right. This is an invitation. If, if you guys see something that that you want to have as part of the trusted software supply chain or the software factory, then uh you know the invitation is that you implement that. I.

F

Just I was making a joke and it said crash loop back off.

I

Yes, yes, like um yeah, not being able to pull certain images. Yes, I've I've seen it all and trying to make this demo but yeah, as you will see like in. uh If we look at the topology view, it's it's a number of open source projects. It's not meant to be just red hat. Some of them will be things that red hat supports again because it's the out working of our consulting arm at the moment, but as andrea says it's we're presenting here, because we want it to be more community.

I

F

This is, this is based on the usdod devsecops framework, which is obviously open source, and they they have all of these design paradigms in there anyway. So really, anybody could take this and apply it to their own system with enough workings. Yes, 100.

I

Excellent anything else before I move on to talk a little bit about the pipeline.

D

H

I'm curious about um uh from a security standpoint, so um I I think I get what you're doing, but um do you like is? Is there security in this? Is there a cryptographic signing of things? Are you using something like in toto underneath or how like what happens if an attacker breaks in- and you know to like this framework here, can they just go and produce whatever they want, as part of it.

I

So I can take a whack at that, though I know we have some of those people closer deploy goes just first. Does anyone closer to plugins want to take a look at it? Otherwise, I'm happy to answer that I'll. Just leave some space.

K

So I'll I'll throw in my two cents in that one. So what I'd say is that logos on its own is not a substitute for any other security controls that you need at each step um and the the work that mark is going to show in terms of the integration with record is, I think, you know, is good stuff in regards to verifying and attesting the output of each stage of the process and the process overall.

K

But you still want to have suitable controls wrapped around it, which is why it also deploys things like single sign-on, for you know, two-factor authentication and things like that as well.

H

Okay, yeah that helps- and I know um in in toto they're, adding like wrecker into the uh latest, like I things like this, so I'm just kind of I was just kind of curious because it seems like it would be almost no work for you all to integrate and we give you like a huge um security differential.

H

um I mean because, like this wouldn't protect against solar winds, uh because solar winds was, you know bad guys getting into the infrastructure and doing things but with in total plus this, then you have at least some hope of that happening and and for you know not a lot of development effort at all, it should be in fact it should be nearly trivial. You could basically be there, so um it was just yeah was just curious about that.

G

I think keylime would also um you know, play a role into this, where you basically start off with boot attestation and and make sure that the the right amount of um you know operating system libraries um boot from from trusted sources as well, and that's, I think, where you where you would start off a 100 secure supply chain.

I

That's right and and yeah, I know in total, it's not the first time. I've heard about it and being close to this project. There's certainly a lot of talk about integrating with it with the north american team. This is just where it stands right now, and key lime and buddha at the station. All that yeah that's yeah can be built. On top of this.

H

Right, yeah, that makes sense, and for those who don't know I'll just say the you know, entoto is basically a way. It's basically designed to take like kind of cryptographic, information about different steps and then let you apply a policy that gets checked over this, and so it's you know. I think, if you put a lot of buzzwords on uh try to distill, what's done here into buzzwords and in total into buzzwords, there's a lot of overlap, but if you actually look at what's happening, there's a lot of difference.

H

So entoto is completely agnostic to everything happening in the system. It doesn't care. If you know it has nothing like the functionality here, and so I think, um there's there's a tremendous potential, because this is a really slick, um really like well done really usable high, like high level system that integrates everything together in a good way, and I think you can just sort of get those security properties from in total, with almost no work and really have uh like the best of all worlds for people using this so um yeah.

H

But this this is really cool. Sorry, uh please go ahead and continue.

I

No problem you'll see more overlap as we go right that there's other places where toto might overlap, but uh but yeah. So that's the platform, as adam said too, not a substitute for typical security controls principle of least privilege. Having you know, this is all set up to use saml and all that other kind of good stuff, um but but yeah. So it's as you say, it's an implementation of uh something that anyone could implement, which is sort of the department of defense's kind of best practices.

I

But there are still things that are left out right. We could still implement, as you say, things like in total um and that's just the platform right. So that's just creating our factory. We haven't yet built anything.

I

Yet when we talk about things like solar wind- yes, maybe we could have had a compromise factory, that's pumping out something that is itself compromised, but what we're going to talk about next is sort of how you might have other controls where you could start to see if something's been kind of tampered with, so the pipeline and the platform, just as we saw uh with the operator right when we were looking at the operator over here, I look at the installed operators.

I

I go down to my plurigus operator, it pipelines and platforms. It wants to think about the platform if you will as the factory and then a pipeline has sort of like one of the assembly lines in that factory.

I

So what does that look like for us? So we already have the platform custom resource. The operator is still running the background, what I'm not showing a ton of- and there are other demos out there that talk about like well. How do I make a project that can that is compatible with ploy ghost the barrier to entry is really low? Nowadays, though, it is still somewhat demonstration when it comes to just any random project out there.

I

There are kind of two things that you need: one is your project should be git ops ready, so uh this pipeline kind of assumes that you're going to have a code repo and some sort of git ops, repo or helm repo in this case right. So that's one thing: you'll see that in the custom resource when we build it. The second thing is that it assumes that you know enough about your use of ploy ghosts that you can kind of see it here.

I

You'll see it a little closer in the demo that you have a jenkins file that tells ploy ghost what version of the the overall kind of groovy script. You'll see that in a second that you want to use for pluto sort of the thing that binds jenkins to what's called step runners. So you may I'll show you, through this diagram and you'll, see in the demo. I make a pipeline custom resource. That's basically a way to say dear plagos set me up.

I

If you will an assembly line in your software factory for this project and you'll see what's in the custom resource, the main thing in the custom resource is what the project is, so what the git repos are and how I want them to be manifest inside of my cluster for us in what we do with our uh our teams. We want to deploy it locally to an in cluster gt.

I

You know sort of a little github that runs inside of our mighty fortress, which is you know, kubernetes openshift, then armed with that, but the jenkins file in the project a build is kicked off just like any other build. It's just ploy goes as part of the platform install the jenkins kind of main server for us that main server is told, and it also gets set up on that main server hey.

I

I have a new assembly line for you, a new conveyor belt, whatever you want to call it for this reference app code project and that looks at the jenkins file, like any other jenkins file in any project which points to a ploy ghost library which binds in this thing, called the ploy ghost step runner, which is basically a way to decouple the tool chain. If you will from what happens in every given step. So what happens in the steps is in this python library, and each step is a conglomeration of these different kind of plugins.

I

So one is for signing. One is for running sonar cube. One is for maven right if that makes any sense again. All this is not necessarily telling everyone in the world. This is how you should do it. This is just so how you could make sense of the demo that I'm about to show you, as the different steps of jenkins, is done, and these are steps that you would recognize from best practices white papers around the world.

I

It uses differently configured steps in the step, runner, library and this config.yml that I call out inside my project. I can further refine on top of what the platform has in terms of configuration.

I

How I want my asset to be built, so some of that information will come from the platform like where the heck is sonar cube in this factory, but what tests I want to run. That might be something that I can configure to the local project right, so the factory plus the project creates, if you will an assembly line which in this case is implemented with jenkins. We also have a techconf label for those who are tech.

I

Town minded so I'll just show you that and then and then kind of questions so to make a pipeline to make a pipeline. It sounds like a novel to make a pipeline, so I'm going to move some things around a little bit here. There's our factory.

I

I want to create myself a new assembly line, so I do that with the pipeline custom resource for our operator and our plugins world, creating a pipeline, I'm just going to do a yaml view of this and I'll paste in my stuff, and what I want to point out is that again I say my overall app is called the reference app and I have a reference app code which is at this git location.

I

Again, who knows where this git location is for the demo it's in github and then I have a helm, a config repo gitopsy thing in reference helm, and I want my service name to be called reference app fruit. It's an it's! A highly trusted app that spits out information about fruit. For some reason, it's a demo so once I create this custom resource, so code helm as we talked about when I go and click create that's going to create that assembly line. So it's going to migrate the projects into getty.

I

It's going to tell jenkins about those projects, it's going to set up a jenkins job. If I look at the pipeline, I can see that hey. It's already done everything that it needed to do to get to the desired state for that custom resource. I can search for jenkins in the in the developer topology view log in using I'm going to use the openshift single sign-on.

I

There is a single sign-on that ploy ghost uses inside the platform, as andreas was talking about, I'm going to look at the blue ocean view of jenkins, and you can see it kicked off a build for me. That was a configuration option.

I

The astute of you may have noticed in the custom resource and it's pulling in a groovy script that builds out the ploy ghost library, so the jenkins file isn't does it has configuration in it that tells how to build out all these stages, which is why you kind of see it built out just looking inside getty here's the code, here's my project, my reference project with the jenkins information right so like I said it points to something that plugo's provides, so that team manages which is sort of the groovy script that gets driven by these parameters, and that builds out a jenkins pipeline.

I

For me, based on this information, so that's one of the things is the jenkins file for my project to qualify to be built in the software factory. The second thing is this config.yml, which is project specific uh options that I want to be able to override in building my asset, and there is one thing that I'm going to override for this demo. Is I've added my own step implementer, which we'll get to at the end this notion of the recore log, but we'll come to that in a second but some foreshadowing for you stick around.

I

If you want to figure out what that wrecker thing is, this is also more foreshadowing. This is the developer perspective on wrecker.

I

I've run it locally within my cluster, but it doesn't have to be run here just for the sake of the demo, we'll again come to that plenty before we're done here, just proving that it's up and running, I can run it locally, we'll talk about what wrecker is why it matters now, while it's busy it's still going to be, it ran those unit tests forever and ever then it here a different, typical blue ocean. I can look at all the logs that'll come into play in a little bit, so there are outputs.

I

Every step has different artifacts that it produces, which we'll get to at the end static analysis. I have output that it produces from from that, and you kind of get the idea it's going to go through all these different stages, it's going to push artifacts to nexus. These are typical things to do for like java projects, and if I just kind of skip through some of this stuff, I create an image. I scan an image.

I

This is the point that andreas made before, where the scan actually broke, and another thing I'll say about having an operator. I had to adjust what tests I wanted ploy ghost to run and I wound up changing a custom resource that the operator managed and the operator wound up blowing away those changes.

I

So if I wasn't, if the operator is running and running properly, it's there are many different controls to kind of keep things from being tampered with, which you saw just go by there, we'll come back to that was sort of it writing out what happened in the build using wrecker or just a demonstration of how one might use wrecker similar to graphios again, as we start to get closer to that, and you can see I'm skipping dev, I'm moving to test and prod. So it's going to deploy my application to test and prod.

I

It runs some selenium tests behind the scene. There's a fair amount of complexity that our consulting team has already put into this, because these are things that they do all the time. This proves that it actually deployed something out to our production. Namespace, that's what we're calling production in this demo and then that's the end of the pipeline all the way. At the end, you can see it's meant to deploy this fruit service again. Thank god.

I

We have a fruit service now deployed and safe, and if I just refresh this just to prove the pipeline is completely finished, and this time it was green, not red, not because I was missing some patch, because I you know as far as this pipeline is concerned, this was except successful assembly run line within my software factory, so I'll pause there any comments, questions things people want to see in the live in the live cluster.

I

It's fine. If not, I won't feel bad.

E

I guess I would like to understand a little bit more about overrides, so in that file there it looks like you have these software factories are docker images. Can you override the arguments? Let's say I have a. um I don't know, for example, um a certain maven java edition that I want to use for the build like 1.8. Can I override that anyway, or it pretty much just comes?

E

What you see is what you get.

I

E

I

Yes, it does I'll I'll talk about my experience and then somebody closer to the project may want to say more. I will say it is 100 possible because I had to do it to make this demo.

I

So if you see that line there, I inserted my own container right and that this is my project with my crappy project, I'm going to put in my own container there, and that was a way I could do it a more a better way to do it right is one that's less intrusive is changing configuration values, as you kind of see here, so I ultimately wanted to adapt what tests I ran with openscap, and I I used you know, sort of their workbench with help from approach shane.

I

I think this on this call like he provided me, this kind of customized version of the workbench to be able to like skip a couple tests, because I didn't want it to test those things because it could fail at any time and that that gives an example of sort of light configuration. What I'm doing here is probably ill-advised and requires deeper knowledge of sort of how polygons is running.

I

I think, most most of the time you'd want to only have to change things like this, but that's how the project will evolve to make stuff like that easier. Anyone who's on the project want to say more about the thoughts behind that, if not that's fine, but just giving space. If anybody else pronounced.

C

Like also applied just, how do you pronounce this.

I

I think it's ploy ghost that's how I've heard it said.

J

F

um Obviously, red hat has a fair bit of experience with um you know: supply chain security I mean red hat, got hacked in 2008 the whole fedora thing. um What was the motivation for this project like I'm, I'm genuinely interested to know that, because to to justin's point there's you know some similar tooling out here that kind of fits into the puzzle. So I'm I'm pretty keen to understand what was the core motivation for building something like this.

I

Again, I could take a swing at that, but I'd rather someone on the project. Maybe.

K

So I'll I'll I'll give it a go. I wasn't there at the start of the project, but, as I understand it, um the primary driver was essentially a a a an identified gap in between here is the usdod's devsecops reference design. Here's the what but not the how and a I guess, a desire to have and essentially something that comes in and fills that.

K

How so that folks, who go hey, I need to be aligned with dead sword, can get off the ground very quickly with a tool chain that aligns directly back to dead sword meets all of the requirements, etc. That's a that's my understanding of essentially why it came into existence, because I looked and went there's nothing that really exists. That makes sense. Have you have you read through the the devsecops framework, the the dead sword.

F

That devotes reference design. I have yeah.

K

Yeah cool cool I.

F

I appreciate that that response I I've I've read through it myself quite a bit and done a little bit with it myself to try and implement you know some of the stuff they have in platform, one for example, and it's very much so this you know massive resource of the you know wild west like where do you kind of start? So I can.

F

I can absolutely appreciate that just just for the dummies, I guess- um and this this might be- you know, feel free to chime in here justin for in total, if it's relevant, but um let's just take a scenario where you know you know um you're you're, a security tool like nmap, and uh you know your host of don.

F

I don't know what, whatever like leonard or something and some big bad cyber gang. um You know roots one of your boxes and you know they pop a vps and they've. You know they've got root and you know all of a sudden they've got access to your software and you know they. They re-upload a version that has the malware hidden in it like. How does this protect users from that from that particular problem? Because I mean that's ultimately, the goal of software supply chain security.

G

Like matt, when you say they upload your software, what what are they uploading like.

F

They're breaking into your giveaway in the past, you download, you know something like nmap right and you have an mp5 checksum you'd. Make sure that the share is valid against what you're actually downloading.

F

um You know if a particular piece of software was compromised and you know an attacker uploaded, a malicious version of that software. You put.

B

It on you know the internet. How does this protect users from that.

K

So kind of, as I was saying before, there is always the need to have additional controls over and on top, but where this can potentially come in is, and I might kind of touched on it is because there is this decoupling between the pipeline and the tools it becomes possible to to write and inject new. I guess implementations of steps in there.

K

So let's say, for example, that we've got a git step implementer, which is basically hey check out the things there's, no reason that couldn't be extended or enhanced to also include verifications of the committer and folks like that. So it doesn't have the tools out of the box today to do it, but it could be built and added in as part of the pipeline.

F

So it's really just a framework to stitch other stuff together.

K

To do something.

I

In an opinionated way: yes, because yeah, if you start from dead sword the what not the how as adam said, we're starting to fill in the how but then questions of like well. How do we really now that we've got a bit of a framework? How do we further harden that framework and how do we make it customizable, because we're trying to balance the interests of security? Well, we want everything to be repeatable, but tool chains tend to be very snowflakey in the real world. So how can we kind of balance those two interests?

I

That's what? If you're kind of seeing emerge here, yeah.

F

I want to share it all with you, man, that's a very fair call: um okay! Well, look it's it's pretty polished so far, so well,.

I

Right polished, yes,.

H

I'm glad it's coming across that back together in a polished way.

I

I

Yeah well, some of that is just open shift. Openshift's didn't know we.

J

Promised this we promised it's not glitch to kind of pick and promise.

E

And um just a quick question in terms of the open source part of this so high level. What I've seen today is there's a few layers, so we have the container. We possibly have the kubernetes deployment and then we have the framework itself, which ones could I go in today and maybe make a pr against? Can I do it at the container level, the framework or all three.

K

I

Yeah go ahead: adam yeah.

K

So everything for this is up. um I was gonna, say upstairs upstream in github, slash, polygons, um all the container definitions. Are there, the workflow definitions. Are there um the operator definition? Is there all written and ansible so yeah? It's it's all there and ready.

F

It's all very biased towards red hat openshift. Are there plans to make the more generic vanilla, kubernetes flavor.

G

We let the community decide. Yes, you know, without for our customers, so obviously we're building on technology. We we know the source code of, and we know it works and um one we in like contributing conscious of time. I really would like us to to demo the the require part of it. That's where the the uh you know the signing and the um you know the attestation basically comes in. If that's okay, man.

I

Yeah, I'm okay with that, but uh yeah. Let me let me get to that before we run out of time and then questions at the at the end so record. So wrecker is a part of six store, so there's a separate group of projects, cosign wrecker, a bunch of things and there's a lot going on in the space there in toto kind of friends uh in this.

I

In this case, what we're looking to demonstrate is just as you guys were pointing out hey this framework is great, but there are all these things you could add to it. We decided like hey, let's try and add something to the framework and recore seemed like a thing to add for multiple reasons which hopefully I'll get to before we're done.

I

So again, what we're doing is trying to write our build activities to a tamper evidence store in a way that may have been something that could have helped in terms of software supply chain attacks like solarwinds. I understand it's, not a remedy for it, but being able to see here were all the steps in the chain that went from this git commit to this container. Build, let's say what, if we had something that could immutably attest everything that happened now, that we have our framework so dead sword the how the framework is the.

I

What now that I've made some opinionated calls about the what it lends itself to say. Well, what? If each of these steps someday had to write out the output of each of these steps into an immutable database, so that auditors or other things, could automatically check that? So what I'm demonstrating here- and this is just a demonstration- this is exactly how it would be implemented. Is I'm saying when I go to sign the image?

I

I'm also going to do two things, I'm going to store the signed image in record to say I, the build chain, I'm going to you know, use my keys to sign it. I the build chain, made this image and then I'm also going to add a node we'll talk about that in a second I'm going to record also in record artifacts that went into building this image and again stored in a mutual database in this case trillium. So transparency.dat sort of google's trillion project open source project we'll get to build nodes in a second.

I

So let me just introduce record really quickly if uh zoom thank you zoom all right here we go so zoom was just preventing me from showing you this video so where we last left our heroes. This is the thing I want to show about signing the container image. So what you see here is I've got private keys that represent the tool chain and it's using those keys to sign images.

I

So I sign a container image which is kind of what you see here, using a ploy ghost's key um for the the factory it signed an image, that's cool that'll be important in a second. It also stores that image a nexus. So it's internal it's using sort of an internal using nexus as a container image registry and then finally, at the very end, this is what I was showing before, where I put two things in when I go to sign the container image I podman sign.

I

I curl push to nexus and I use my recore log. I log to my local kubernetes local kind of recore instance. I log two entries one is sort of the last build node we'll talk about what that means in a second and there'll, be some command line here. So you see the record url. Has a service level address service local address? I need to turn that into a public address, just to prove that this is my record server. I have it exposed publicly again think of record how we're using it right now.

I

Is it's a little bit like graphis? I don't know tons about graphics, but it's meant to kind of do that graphics, meets trillion. Let's say and it'll become more apparent as we go on. If I actually look at the entry that would the build chain wrote to record what I say is: oh my god jason's.

I

Let me just make it a little prettier there's a body of uu encoded data, but there's an inclusion proof as you'd expect from immutable databases that says okay, so I've got a database of a there's, a merkle tree behind the scenes. The body is interesting, we'll get to that. The thing at the top is the uuid of the leaf node, which wrecker uses to find entries.

I

If I look at the record command line, there are things I can do with entries and record. I can verify.

A

I

Entries merkle root and collusion proof all that I can get those entries I can search through those entries, and I can upload uploading- has been done by the by the build chain.

I

Now, if I go to the next, I think I'll just jump to the next one. This is going to talk about build nodes so and again the questions right at the end, just for the sake of time uh so adam and I got to chat about this. This was kind of fun. So here's how we imagined that something like this could work both for attestation but auditors, and all that and also eventually you'll, see at the very end, a way we can hook into.

I

Maybe things like entoto's doing or kritis does or all this kind of stuff. I built an image with a tag you could imagine. I could build an image and make an alias of that tag as the uuid. That represents the last thing that happened in the build chain.

I

So that last thing is the is sort of my first build node that first build node could say here's the step so that first build node is sort of saying this is the culmination of the whole build chain I built artifact x and the previous.

I

If you want to see what happened before, here's the previous entry in record, which is itself a build node, so sort of a linked list inside an immutable database, and this thing would be all hashed in the same way, you'd expect from a mobile database with its step name with whatever output is relevant to it. Maybe it's the unit tests or you know, pick a thing or maybe it's a static analysis with a entry to the previous step and so on.

I

All the way down to eventually the previous uid would be zero and thus you'd have a way of creating the whole provenance. Everything that happened from checkout from debt all the way to the creation of an image in an immutable database that you could verify for yourself. So anyone anywhere could look this up as well as for the sake of auditing. All this kind of stuff right so again, let me run through this and then we'll get to questions at the end. Just to give you more a sense of what rekkor can do.

I

I can look up the last entry in record by saying get me that uuid get it in json and use jq just to kind of format it, so we can make sense of it. This is how rekkor stores things in trillion right. So it's got a body. You can change how record stores things that's outside the scope of this, but the things we care about in its data is this extra data which we'll get to the signature is stored in wrecker. The public key is stored in wrecker as well as uuid.

I

Wrecker ultimately wants to be also a pki solution, because with this with the immutable database, I don't need to keep those keys forever. If I can prove I had access to those keys at the time I sunk it into the immutable database, but see sigstor the record project to learn more about that.

I

I'm going to look at that extra data to pull out my build node. When I pull out my build node, I see that the step in question again for the demo there's only one step, that's signing anything! It's that sign container image, step, which you see there right so in this step and that matches that string. Exactly that's what I wrote out, I said my step was sign container image.

I

My step output was base64 encoded, but we will see in a second that that is exactly the signature that was stored and then finally, the wrecker id. So what I've done is I put into record proof that if the image was signed by rekkor now this whole build chain this chain of of what happened bill of goods on the assembly line is now all signed in record. So I have another way of verifying that this the build chain did what I thought it would do in terms of verifying there's a number of different ways.

I

I can verify I may skip through some of this. You can verify by artifact. You can verify by public key right. So if you only have the artifact or you only have the public key, you need the artifact, the public key and signature. If you didn't have the original entry, that's kind of what I'm going to show here for the sake of time, I'll, let it go for a second just to give you a sense. The artifact, in our case is the last build node.

I

The last signature entry. I'm going to pull that out of record. I'm going to look to record to get that. I could get that somewhere else if I wanted to, but I'll pull that out of record. So that's the detached signature and the last bit is a public key and just showing that I could get that from anywhere.

I

If I had the public key in a secret and again I'm just using the open shift ui to quickly navigate to that secret right and again demonstration, I probably wouldn't keep the private key there, but if the public key is right there, I can say: hey lego's public key, I'm going to paste that in and now what wrecker can do for me is not only verify the inclusion proof but verify that this artifact was definitely signed with this public key, and this is the signature that goes with that artifact, and it will do all of that, and it says here: yes, it was otherwise it wouldn't have returned at all.

I

So it says what the hash is. It gives me the tree root in case. I want to do my own kind of inclusion proof and in fact the cli does its own inclusion proof locally, based on the sha's from the tree. Current tree size is only two because I've only put two things in, because this is a demo, and this is the first time I ever ran this you can see.

I

I was this is the calls that I was making to my local record again, not so not so interesting, just proving that I was calling wrecker inside of here there's also a public wrecker instance, which I chose not to sully with my demo.

I

You can also search through wrecker again just to give you a sense of what record does I can search by public key or by sha? So if I want to take, if I want to take my artifact, which is the build node or in this case you know that build node that wraps the signature, I could search for it that way, search by shaw, which is what you kind of see here and I get back the uuid.

I

I expect that ui uid again for the second time I'm going to skip through that the uid does match what I had before. I can also search based on the public key, so I could take the plugins public key and say hey. Let me see all the entries that polygons has made and there's two our build.

I

Our link list is only two: it's the one at the end with the container signature and then there's another one which has the output of the build right and that's sort of what you see here, just proving that there is a public record server out there there's no reason it has to be local to my cluster. In fact, we want it to be transparent, but just proving I didn't write to that record server and I may have the you know the tree root changed. I could know that. Oh this isn't the tree.

I

What I expected for anyone who's verifying this and then. Finally, this is the last bit. I wanted to show you guys today, let's play with these kind of build nodes, I told you the last build node is what we said was the signature in our case it'll be the signature of the output, the container output right. So, let's just see, if that matches, can I take a public key, which is the ploy goes public key? I have it locally here.

I

Can I basically get that signature file right, like just proving at the signature file, that this thing is signed by ploygos when I then go to decrypt it with gpg? Right I download it, and then I jq, what I should see is sure enough. It was signed by the service account again the tree again, a demo. We don't have the verification of this key, but we know it was this key.

I

Regardless of who owns this key, I can also do the same thing with the signature content that was in the last build node, and using that I should be able to decrypt.

I

I should be able to use the signature content that signature content should represent the signature that was in nexus blah blah blah. If I output that what we'll say is should match this appear, which it does see, it's the same thing, it's the same build. So these are ways that you can start to do some verification.

I

The other thing I want to show is: if I go down through my link list in the tree, I can get to that previous build node. Just by looking at the last build node pulling out the previous id and again you can imagine wrapping this in some sort of cli.

I

I can get that previous build node and then look at its extra data, which will show me the step name, some content and the previous record id, and that's what you see here, although the step is much bigger, again same step, because it's the same step that I ran this from sign container image. But that represents this first thing that I put sorry yeah yeah this first one that I put right over here.

I

I highlighted the wrong one, this first one which is the build output right, and so, if I decode that which I'm going to quickly just do right now, following the same things previous record id 0, that means there's no further entries in this chain. If I take the step output, you see the step results that from this build. So this is just an xml file that gets produced by ploygos.

I

Again, this is just a demonstration, but it kind of gives you an idea of the kind of things you could have per step and be able to unpack, and that's where I want to start.

I

The last thing I'd say before we end our time is a future demo, and I think there are other people trying to do this, but you can imagine I didn't get time to do this, integrating with things like opa or gatekeeper, which brings opa to kubernetes you're, going to imagine a world where even a simple admission hook, that does a record verify on a uuid of an image in certain participating. Namespaces would allow you to follow this pattern that you see from the transparency.dev website about assume this is kubernetes.

I

This is the deployment you could have the web controller kind of checking. Oh hey is this an image that I know has verifiably been built by my tool chain, so possibilities for the future.

G

I

G

Open policy agent, just in case you.

I

That's correct at the end we kind of threw a lot in all at the end. Sorry, we probably left too much time for questions in the middle, but yes so questions now for the last three minutes that we.

G

Have or anything that you guys want to do sorry.

I

G

You off before matt, but I I thought it's important to um no.

B

F

No, no thanks so much guys. That was amazing. I'm that's a cool project.

E

Yeah, I like it because I'm I'm sort of building like a common patterns like with myself, and it's like. Why am I doing it by myself? I can contribute to something like this right.

E

So is it like a slight group or something where you can be more involved?

E

How do we what's the next step? How do we, how do we engage.

I

That's a great point: there is one for record. There's a slack group, I'm not sure there's a slack group for ploy goes yet adam. Do you have more there's.

K

No slight group for polygons yet, but to be fair, we probably do need to.

I

K

At the moment, the best place to engage.

G

Is on on the github.

K

Repos yeah yeah, yeah, okay, cool awesome.

G

So shame that we are out of time, but I did want to get justin's view as well. In terms of you know, uh solar, wind and and supply chain attacks and and how that would help.

G

So, ideally you build your your employees um operator as well through a software uh you know to the to the factory, um but the the point is: where do you start right and would that ever be possible without actually having full control and understanding of the source code that is used to to build whatever it is, whether it's the operator or the the software right? So matt was saying what, if someone um uploads, you know dodgy sort of components, and I think the the records or merkle tree um addressed this a little bit.

G

But where do you realistically start.

H

Right so yeah and in toto has different ways of managing and handling this. We actually worked with git and so they've redone parts of their signing scheme, because we found design flaws in the way that git signing worked for git tags and other aspects like that. They actually use santiago's code who's, the lead of the intoto project and a design that he and I and some of our collaborators came up with so yeah. There's, there's like a bunch of stuff that exists.

H

That already does this in the like in totoscope, because I think that project's been working at this problem from the different angle, where we've been very security focused um from day one and that's been really the sort of primary thing, and also we've been, I think, very vendor and technology agnostic, um and so like I'm really impressed by what you have and really um I'm looking forward to digging deeper into.

H

uh You know like some of the different pieces, but it's also a little in some ways hard for me to understand, because I don't have the there's. There feels like there's a lot. That's very vendor specific here um and it's hard for me to disentangle some of the security properties from you know some of the the other things going on, but uh yeah I it was really enlightening and really enjoyed learning about it.

G

Thank you, justin matt. Do you want to close the call or are there any further questions.

L

Yeah sure sure did anybody else have any other questions.

L

No thank you very much. That was. um It was awesome, great great! Well, um look uh if you guys want to get in touch.

F

With andreas he's, he's on slack um so feel free to reach out to him, you can check out the github repo and thanks so much everyone from red hat. That was that was really enlightening. I really appreciated that yeah thanks so much.

G

That was really cool awesome thanks for having us and see you at the project. And yes, if you have anything um in terms of extensions that makes it you know just as applicable for any other platform, absolutely right. So that's the exactly the point behind an open source project. So thanks a lot for having us.

F

Yes, yeah, I think look as I mentioned. I took a quick look on the call at the repo. um You know. Obviously you know just available for openshift right now, but from what I read, it doesn't seem too hard to get this running on vanilla kubernetes. So I'm gonna gonna have a little bit more of a play and yeah thanks for sharing hey um jj. Are you still here, buddy.

M

Yeah thanks thanks for putting this together. This is an awesome project. One thing I would suggest is try and see if we can pull together a demo that has uh that doesn't have anything, that's uh open shift or red hat specific.

M

uh It will serve a few purpose in terms of trying to get wider feedback and adoption uh and for a project. That's as useful as this. I think uh it'll also be useful to see how this plays in well with the other open projects that we have like what justin was saying about in toto and stuff. So I would uh I mean, if you're interested and if you're curious in uh getting more community involvement. I think it'll be a useful thing to do. That's that's not uh a demo! That's not too underspecific!

M

That's what I would do, but uh otherwise it's a use. It's awesome! It's it's a good learning for a lot of a lot of folks and uh I'm pretty sure it's going to be useful for uh for the community overall.

M

Thank you so much for putting this together.

I

J

Thanks for the feedback, we're definitely going to take that into account. You know and hope, to come back in the future, obviously to show how the project has evolved. Awesome.

M

D

Great um is anybody else on the call, wanna wanna um say anything before we? We close, I think, we're pretty much running out of time now so.

E

Yeah, I guess just what we briefly touched on at the start, um I'm going to keep going yeah like if anyone wants to maybe hang out for 10 minutes or something um sometime. Just just. Let me know.

F

Yeah for sure that.

E

F

E

I have a discussion on one of the key talks that we go on, or maybe at the end of it we can also do key takeaways and have a talk about it or something like that. It just makes it um nicer being not in that region. You know, because it's quite an exciting event for me, so it'd be nice to to share that with people and talk about it.

B

E

F

Cool all right, well, um cool cool! Well, hey look um just for for those of you in in you know the in in australia um or or even apac.

F

I guess it's pretty relevant um just just to let you know uh brad and I have have been in touch with bill mulligan from the linux foundation and um we're going to be spinning up kubernetes, like the kcds, the kubernetes community days here in.

B

F

Where you know we're reaching out to different vendors for for sponsorship, etc at the moment to try and get things organized um there's a very open invitation to anyone who gets wants to get involved in in something like this. um It's you know a massive undertaking um for full transparency.

F

The the kubernetes um forum 2019., like today, the largest conference, that's been hosted in australia for in the cloud native space that that's been indefinitely cancelled um and and it's the view of you know the cncf and linux foundation that this will kind of, replace that now um so yeah. Look if you want to know any more information or if you want to get involved. Please just ping me because brad and I would really appreciate any help or support you could offer.

H

F

um Yeah other than that, we'll keep you guys posted and yeah all the best we'll chat to you guys soon. I guess have a great day.

E

Cheers tim cheers dear my boy yeah thanks.

I