Red Hat OpenShift APAC Hybrid Cloud Kopi Hour, 27 Sep 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: APAC Hybrid Cloud Kopi Hour (E8) | Securing the Software Supply Chain

Description

This week we’re again joined by the Red Hat Demo Gurus! Stop by for a chat with Prakhar and Tyrell to take us on a journey into the exciting world of Software Supply Chain Security. Following up from their excellent blog at https://www.redhat.com/en/blog/improving-containerization-security the team will discuss the why and how of this important topic and give us a real world demonstration of some of the cool products and tools Red Hat has to offer in this space.

Slides: https://speakerdeck.com/redhatlivestreaming/apac-hybrid-cloud-kopi-hour-e8-securing-the-software-supply-chain

00:00 Stream begins
06:43 Securing the software supply chain overview
20:00 Red Hat Trusted Software Supply Chain
37:15 Demo!
01:02:26 Closing remarks

A

A

Hello and welcome to the latest uh APAC hybrid Cloud copy our super exciting to be back here, uh another where I think it's only been a couple of weeks since the last one. So it's super cool to be here as always, I like to introduce the fact that we are the Kopi hour, uh which is uh this wonderful, coffee beverage. You can get in Singapore um that I do so love.

A

My name is August simonelli I am a technical marketing manager here at red hat and I am so so excited to be rejoined by my co-host. Extraordinaire Dev Sean bag uh dev's been away on vacation. uh So no, it's really great to have you back how you been Dev, say hello,.

B

Hey August um really good to be back and you know, joined in the show. Looking forward to another awesome episode of the copy hour, cheers to you and um you know, just uh have to have the ability to have Rucker and Tyrell present um some really cool stuff with us today. So yeah over to you.

A

Yeah definitely so we met prakar last week, um so prakar sivastava uh go ahead and say hi again, um yeah just introduce yourself yeah.

C

Thanks so obviously we met in our last episode where we talked about uh event driven intervals, so today we will be focusing on something related to security.

C

So, as my introduction, I'm working as a team lead principal architect for uh we had demo platform team, there's the t-shirt which I'm wearing today for that we showcase the capability of of all our products by aligning them with the marketing messages and assist our sales by developing the demos for them. So they are, we basically develop the sales assets so yeah. That's what we're doing that's.

A

Right and uh we're also joined today by Tyrell Reddy, who is on um on prakar's team, uh and so that's super exciting because we're getting like all our gurus in one on one shot like last week, Tyrell say: hi hi,.

D

Everyone, so my name is Tyrell I'm with prakar's Team uh he's mentioned most of what we've done to already so I won't bore you uh yeah, so we within the demo platform team- and we are just going to show you one of our demos that we built today here over.

A

To you, that's awesome, so I noticed you, you've got the shirt on as well so and Dev is wearing his his red hat shirt. Now I decided to go bold here and I've got. Excuse me. I've got my open stack shirt on that's right! That's right! So that's that's! Right! I've got the openstack shirt.

C

On yeah, open stack on openshift so well.

A

There you go, there's a future show. Do.

C

You have all this time. We have open shift.

A

So yeah, so we have all the stacks and shifts.

C

Oh, it was wonderful, but yeah, I I.

A

Decided I found that in the closet and I thought you know what that's going on so before we kick off I want to. uh In our uh description for the episode we teased an article that uh you guys wrote and I wanted to call out a few things that I saw in there that I found really interesting, so I've put together. One of my my little slide, Decks that I wanted to do.

A

um I want to talk about some of the numbers I saw in there and they're numbers that both EU, Tyrell and picar will recognize so, but I kind of wanted to highlight them and and um sort of address what they mean. So the first one is 49.. uh This one jumped out to me: what is it 49? Well, that's simple!

A

That's the number of vulnerabilities that you guys say uh was from the study that's listed in the in the doc in the average application 49, that's that's pretty rough and that you also mentioned that there's 80 direct uh dependencies in in those applications which is like you start to realize the complexity of what developers are dealing with when they're trying to actually create this stuff. So I continued reading and I thought you know. That's that's scary and the next number that popped out was 4.45 and oh there's a lower number. You know.

A

Maybe it's not as bad problem is. You guys mentioned that this is the number of million US dollars that the average total cost is associated with the data breach. That's scary, right I mean we're not talking couch change. You know we're talking a problem here and in an area where the complexity in our in software has has created a financial issue that needs to be addressed so I kept reading and I'm thinking. Well, okay, here's another number 83.

A

I, like that number uh I love the number eight but then I read on and I learned that it's 83 of organizations have experienced more than one data bridge and I'm, not totally surprised by that. But it is like you really start to think oof. You know. That's that's not good right, so you can kind of see that the Doom and Gloom story is mounting. I thought. I gotta find something good in this.

A

So here's something good, um the the there's, something that's Rising and then I looked closer and I read what you guys said, and it said that the study also found that the rise of artificial intelligence is further making this uh complicated. It's increasing security breaches yeah. So we've got all this stuff. It's costing money. There's lots of exploits everyone's being data breaches.

A

You know then I then I saw this 70, 20. and I know well, they're, looking at that and they're going 70 20.. What is that? Well this? This is probably one of the harder ones and I really wanted to call this out and that's how badly my team lost this week. So yeah I'm a fan of the Denver Broncos in the National Football League. It actually got destroyed by Miami and so I I felt. You know we needed to to bring in if we're talking, Doom and Gloom I'm, bringing all the Doom Gloom in.

A

But you know what I'm not worried because who do I have on this call. I got the heroes we need to fix this I, don't know if you can fix the Broncos I love. I would love it it's good, uh but what I do know is that you guys have come to tell us a little bit about how we can deal with the scary situation that is going on uh for security and and the products that red hat and- and you know, basically, the open source Community has to address this kind of stuff.

A

I will actually um share the the blog post address there for folks you to take a look at, um but with that intro I'd really like to kind of learn more from you guys about about how uh we're addressing this at Red Hat cool.

C

Thank you see, so, let's uh I just want to show some slides uh with some more numbers. So let's, let's talk about that? Yes,.

A

And I I tried not to take any of these numbers and I'm sure you don't have the Broncos in there, but yeah there's, there's plenty more good numbers. Yeah.

C

Okay, uh we have a big agenda today, which we will be talking about so I'll just start with the stuff which sorry yeah so we'll start with. What is software supply chain? Security? Okay, so we we uh and why it is very critical, like August, has just thrown some numbers that uh 80 of organizations have already faced some kind of bridge.

C

Now most of these bridges are due to the dependencies which uh the libraries which they are using the sources from where they are getting those those libraries and they don't know exactly where those libraries are coming from, especially when the companies are more and more relying on the open source softwares, where they don't exactly know which source from where that particular library is coming from that place of software supply chain. Security becomes really really really really really critical.

C

So we'll start from there, then uh there's an executive order uh by U.S government for the software supply chain, we'll talk about that, what they want to do and what we can learn from that particular uh in executive order. Then we'll start talking about bit of the developer flow so how the developers work in uh um and then we will talk about the shift left strategy which is Again part of the developer flow, so how we bring Security in the early stages of development cycle.

C

So this is something which we'll talk about we'll talk about the code built and monitor. How are the tools which we are using and which tutorial will be showcasing in the demo? We is to protect our softwares and our customers can learn from out of that. I'll talk about the bit of the high level architecture, what we have built as a for this particular demo and which clusters are running, what tools which we are having in on on which cluster we are running it on and then title will showcase the whole demo for you.

C

So let's start from what is software supply chain so.

C

C

Okay, okay, so uh I'll start with the. When we talk about surface of software supply chain security, we always uh discuss about devops desktops, uh so starting with devops or devsecops, which is basically devops on steroid or devops, which is boosted by the security. It is blending the teamwork in automation, development with all the security practices. So this is one way of looking at the security when we are bringing security into the uh in into the developers development cycle. And what comes? What is software supply chain?

C

In that case, then, the software supply chain is basically it's about ensuring each and every software component, which our customers, which organizations are using and to check their authenticity to verify that they are safe to use, verify which source they are coming from, how to and once the they write. Once our developers write the code. We make sure that that particular code has been gone through some scooting they have been tested.

C

They have been signed by that particular person who have done that once we build up the images, uh we know that that particular image was built on a authentic based image. Maybe the universal based image which redhack provides, which has been already been tested for all the security measures and then after they use that as a base image, they put their applications on top of it, then we sign those particular things. So that is what all about the software supply chain security that every aspect of your development cycle.

C

We built Security on top of it, let's Okay, so that was uh the devsecops and the both the concept which I was talking about, where we are combining the principle of devops and also the security which is we are boosting. Its security and software supply chain is basically looking at each and every software component which we are having and especially when you are using the external libraries.

C

The the thing which is we have to think is every organization in current world is somewhere using a OSS or open source software, and they don't know from which particular Source they are downloading. They can download a simple. A developer can start writing a code by just downloading a binary from internet, and he doesn't know why it's been compromised by somebody or it's been already been. uh Have some malware into it? How to translate? So that's what software security is about.

A

And it's a good point right because you think about even the way like I'm, not a developer but as I learned and I'm cutting and pasting code and I, I, Google and I find that something is. You know.

D

A

How do I do something in in you know, JavaScript or something and I quickly and I cut and paste and I copy. You know, I, don't check all these things, and, and so with all this great open knowledge, which is great, you know, mistakes happen even if someone's not trying to be nasty. We want to find things that aren't right. So you start to realize how complex this can get.

C

Yes, yes, yes, true true, so let's talk about why this software supply chain as August, was saying that we don't really sometimes care where from where which source we are just getting that particular library from so why it is important. Now when we say supply chain attacks, it's about when they will happen, not if they will happen. Okay, so it's it's!

C

There are all all the numbers which August had thrown in front of us shows that. How much is how much security attacks are happening? How much dollars we are losing like 4.45 million dollars in 2023, which was in 2002. It was 4.36, which is now 2.6 percent increase in that particular attack. So we have already lost that. So this is something which we have to take care, even after paying the ransom. The real cost is the downtime in the recovery. So, okay, what what happens when we, when we are attacked for ransom?

C

Okay, we pay for ransom, but what about the downtime? What about the recovery from the data bridge and how? How will you know that after even paying the ransom they have provided all they have not? They are not releasing that all the data is of your customers, so very difficult to that's why the software supply chain security is very important for us now. Here are some more stats, more numbers. uh There has been 74 742 percent average annual increase in these attacks over the past three years, okay, which is a big big number and 20.

B

C

These data breaches were traced back to the compromise software supply chain and because our developers, they just use a library which they don't know from where it is so that's like 20 of those breaches were just because of those compromise. Software supply chain.

C

Eight percent of the organizations are looking for collaborating the devops and security teams. Together they want to work together. They want to find out that what can be done to protect from these attacks and say 92 percent of organizations believe that all the OSS Solutions are privately for the transitions to moving to the open, hybrid cloud. Okay. So that is something which we need to look at. That's the good news for us like people us have started thinking about it. People started talking about it, so at least, let's see how we apply all these things.

A

Right and we're not calling into question open source. It's not like people are saying. Well we're not going to bother with open source. It's it's like we. We need this, it's a proven model. It works. Let's, let's make it better.

C

Let's make it better: yep Okay, so no numbers, no okay, the growing.

A

C

So, let's, let's talk about some attacks, which happens like our uh digital landscape is increasing. We are getting AI. We are talking about chat DPT. We are sending all those messages, uh so threats are also increasing. We are seeing security so now, security supply chain is not just a buzzword. It is crucial for protecting data, intellectual property, Source codes and, let's quickly highlight some of these scripts like stolen certificates, still a very significant threat.

C

People steal the certificates, type typo squatting, typo squirting attack a simple instead of typing, maybe uh Bank of America, you just say Bank, oh America, simple, and then you go reach to some place where your attacker is sitting out there and looking for your username and password dependency confusions a lot of the times.

C

They are ambiguous dependency dependencies, which people use, which can be exploited from compromise built environments like I, was saying that when you are using your build environments like a base image and uh instead of using it coming from a trusted Source like Ubi image from Red Hat you're, using somewhere from some other third-party Registries you're downloading a based image, and you don't know what that image has a pre-installed malware so risk before we begin our development.

C

So before we start building our development before we start writing our first line of code, you already have a malware installed in your base, images and or any malicious from your course. So these are subtle but deeply harmful for all these attacks, so they all so that's something which we need to take care and where our software supply chain security comes in to rescue this.

B

Is quite an exhausting list, you know it's kind of covering. You know the source, the packaging, the build part right, I mean I was just thinking. You know when and I don't blame the developers. You know when you do an npm install it has. You know 50 to 500. You know dependencies. Yes,.

C

B

Is easier for the developer to you know get to a particular point, but you know how much do they you know rely on this. You know all these dependencies and whether they are protected. They are secure that comes into question so really interesting. Yeah.

C

Yeah, that's what what we are trying to bring in is uh developer needs to understand from where that particular library or dependencies is downloading, and also when the auditing team comes in. If there's an attack happens like it's very diff, you can't stop it. You can prevent the attacks. You can slow them down, but you can't just stop those attacks, but at least if we know the reason of the attack next time, it will not happen. So so that's something which we are looking at in the secure supply chain. So.

B

What you're trying to say is it's always evolving. Yes and.

C

A

Think yeah it's it's quite a I mean that's that's a great call out Deb is that it's complicated and evolving so I mean I. Haven't heard of half these things and I know what they are like. You know, when you explain them, you know the typo squatting with a great name but yeah.

A

You know we do this show in in in a month in in three months, and that list has changed. It is longer and so yeah it's it's insane. Yeah.

C

Okay, so it's not just us who are worried. Even the governments around the world are worried. So again the U.S had just done a executive order, which is 14028, which is enhancing the cyber security. It is basically a security standard for software purchased by the US government.

C

Nist is the key player which is charged to boost the security measures. They are creating guidelines for mainly for U.S uh agencies, but we can learn from the same standards that how we can make our software supply chain more secure.

C

So we are just learning from those standards which nist is trying to make as well, and our aim is to uh if, uh if the viewers who are listening to our live streaming, if you are guys, are working with us uh agencies, if you're working on the projects which are best for the use of U.S federal, especially you need to read about this executive order. You need to. You will find out the framework, how the the Cyber, uh how you can secure your software supply chain. So that's something which is very interesting. Go through.

C

It read through it uh some of those good things we learned uh in red hat and we are trying to implement that in our a secure supply chain approach.

C

Okay so after what is the red hat take on crusted software supply chain, what we do the first of all? First of all, uh over the three decades, we are Distributing the software we learned from the best how to distribute those office. Now everybody knows that we have. uh We had package managers which is RPMs, which we are Distributing to our customers, using Red, Hat, CDN content distribution, Network, how we ensure that those packages when they reach to the our customers they are reaching in uh they are coming from a trusted source.

C

So what we do? We sign those RPM packages with gpg keys and we provide the gpg keys to our customers so before they install those packages they look, they verify those packages with our gpg signatures we have, uh then we uh we started working with the disconnected environment. We got satellite server. We asked them that. Okay, if you are in a disconnected environment, you can connect through to the red hack, CDN from Red Hat of satellite, and you can locally get those packages from satellite. Now, satellite, not first, not just to distribute the packages.

C

We also send notifications. We send if there's a new uh cve um comes in and we patch one of our applications. We immediately report it and satellite itself will show you that these are the servers which are basically uh affected by that particular cve, and there is an automated way of packing all those servers using satellite. So we have. We have uh implemented all these security measures in red hat from and built them from last 30 years. We clone all the code uh from to our internal repositories.

C

First, for the control, our distribution is rock solid. With the by the signed packages, I was explaining. We have implemented the robust safeguards against any tempering. So if anybody tries to put any malicious code in our RPM, those thick Natures will say. Oh this particular RPM package is not signed by Red Hat. It is coming from somewhere else, so I'll not allow you to install it, and we maintained our products with minimal changes, ensuring that production against any unpredictable code risks from the Upstream.

C

So we try to do all these things and now what we want to do is we want all this stuff, which we have learned in last. Close close 30 years, we want to bring that and make our customers to learn that from us. We want them to share out the knowledge which we have gained in last 30 years to distribute those packages. We want to share uh the how we do it in internally and then how our customers can benefit from our experience. So this is what we are trying to do here.

A

And I think this also highlights that, for for especially for Red Hat security is not the end of the story, it's not like we, we build something and we test it and then we deploy it and then we secure it. It's it's more like we secure and then we build and we secure and we test and we secure it like it's happening at every stage and I think what you'll see with the demo and with the car keeps going. Is that yeah? It's that's where we're extending this.

A

Like he's just said into that next space, every.

B

A

Of the way there's secure, secure, secure, secure it's it's, not it's not sequential, it's there, it's just there. It's not even you know, rinse and repeat it's just always part of it. Yeah.

C

Okay, so let's, let's take you towards the stuff which we want to do, show you in the demo which channel is going to Showcase so uh before a title starts. I just want to bring few things in here.

C

um First is the developer flow now developer always operate uh in within two distinct Cycles, whether it's the inner loop and the outer loop. Now most of the magic happens in the Inner Loop, where concoding immediate testing debugging it's where the developers are having their immediate feedback loop, and then we have the outer loop, which encompasses uh from integration testing to deployment to security monitoring in production, so that, of course, in the uh outer loop. Now, if you see here in the outer loop, the security is in the outer loop, where security week.

C

Traditionally we keep security at the end of the development cycle, which we want to change. We want to bring the security from the point when the developer starts: writing their code from their IDE from their favorite vs code or from their favorite uh Nano or VI or whatever they want to use as their IE. So they want to bring the security in there. So that's something which we want to talk about in in today's session.

C

haha Now, one of the major things to bring Security in the Inner Loop is the shift lift strategy. Now, when we are discussing the developer flow, we look at the shift left strategy, which represents the transformative shift in our approach to security Now. Rather than addressing the security at the end of the development process. We want to bring the security in the starting as I was talking about. As soon as the developer writes his first line of code, we want to bring the security Now. This strategy naturally complements our inner loop.

C

We are coding, testing and debugging is taking place now by shifting security left. We embed the security considerations directly into this initial Loop. Now the outcome potential vulnerabilities are identified and addressed much earlier in the development cycle, or we just don't wait the product to be out. We are not waiting for that. So, in essence, shift shift left, Champions a proactive approach, embedding the security away development right from the beginning. So that's something which we want to showcase today.

C

Okay, now catching dependencies and availability is early, so core integration with application security systems, as I was talking about now. One of the aspects of our approach is focusing on the transitive dependencies. There are indirect dependencies in the software. They can be potential security gaps like from which libraries which dependencies we are using now imagine as a developer when you're writing the code uh in the film in your IDE and there. If you get immediate feedback. Oh the library which you use is having this particular vulnerability.

C

You can immediately say: oh I, don't have to use this particular Library. Let me get the library which doesn't have this particular availability. So that's what we want for you. So as soon as you said, the developer writes this code. You will find out. He can scan his code. He can figure it out that all the stuff which he had used in his application is vulnerability free. There is no vulnerability inside it, so this is what we are trying to bring with our secure supply chain approach.

C

So we are not just waiting for the Post development scan. We are proactively flagging. Real-Time issues in the developers workspace so when developers are writing their code on their laptops, they know that that the code which they want to push into their uh repositories into their scms, they they are free.

A

With any of the vulnerability.

C

So that's what we are trying to do here.

C

Now build with security, focused CI, CD, workflows, Now security drives our our CI CD process, August 1st year of explaining that in Grid hat uh we don't have security. At the end, we every stage of the building process has security aspect attached to it. So that is the thing which we learned in our uh development cycle. Now we want to bring it to our customers so that they can build their CI CD workflows with security Focus.

C

Now, safety checks are everywhere in our pipelines. So one of the things which we want to do with the security is bringing s form or a software bill of materials. A software bill of materials uh bringing the software bill of material is, is basically it is the list of all the softwares or of the all the ingredients which you are bringing in, uh which you are using to uh write your applications.

C

It did it clearly details all the components in the build allowing for the transparency allowing for easy tracking of each element, and this also makes sure that it is easier to identify that uh the potential vulnerabilities. So that is one thing which we when we are building our code, we generate a s bond, which is software bill of materials which will have the list of all the libraries which have been used which source they are coming from. Who is the author of that particular Library? You will have all that information.

C

Now it is as a part of the executive order. It is part of the compliance for every software as bomb needs to be generated uh before they gets deployed, and also it can. uh If you have list of all the libraries all the authors, everything in front of you, you can your auditor teams, your security team, can easily figure it out. Oh, this particular library has some vulnerabilities, don't use that use this Library, which is coming from a trusted space, and they know that particular part as well.

C

Now another thing is origin checks. Do you want to ensure that we know the source, uh the source of the software which they are using? We want to know the source of the build images which they are using. uh We can build up the deployment rules uh which, with the secure guidelines, we can continuously scan the codes we can sign and Trust. So uh Red Hat has been working with six tool and uh a little bit of a sixth row and bringing something which uh for signing okay.

C

Now we can sign our code so as soon as the developer pushes his code into um his sem into git repositories or whichever sem uh you're using. You can sign that course, so that if somebody looks at the code, he knows this developer had written the code and it has been signed.

C

We want to bring the signatures for distribution of the images so after that particular code is built and we want to and we want to distribute the images uh after we build that those images we want to sign those images using cosine again, which is part of openshift pipelines using tecton chains, and we are working with sixth floor cosine and we are supporting them um and we are using that particular tool to sign our images for distribution.

C

So, if you remember, when I was talking about it had 30 years, we were signing it with gpg signatures. Now we are using the same concept to sign our container images using 6.45. So that's the experience which we learned. That's what we have learned, how to distribute our RPM packages. Now we are bringing that to our customers you're going to distribute to images you want to keep them secure. You want to know from which authentic Source they are coming from after building sign those images and then distribute it and.

C

And we also are bringing the safety gates, which has barriers, ensuring that the secure and quality work only progresses to our uh plus those two hour. Applications which are going into Productions.

C

Now, uh okay, we have done the coding. We have signed our code, we have looked for the vulnerabilities. We have built our images we had verified. All the images are good. We assign them now what now we? Our applications are running in Productions. What next now it comes to monitor them so applications which are already running in our environments. We want to monitor those runtime applications.

C

We want to know the real time vulnerabilities, because every day a new vulnerability is going to come every day, a new, a common, a cve will be released, so how we will know that our running applications are not getting affected with them. So we have monitoring real-time monitoring, so our security doesn't just is for and post deployment. We will be doing the continuous security monitoring. We will always look out for anything which is uh any software piece of software, which has a vulnerability during the runtime of our applications.

C

So for that we have a tool that had ACS Advanced cluster security, which uh title will be showcasing uh in his demo and he'll show you that uh how ACS is not just helping to scan the images to find the vendibilities during the deployment time? Also, he will show you the advanced application is already running, then how ACS is going to scan uh we'll look for any wonderful new vulnerabilities which are coming in again. Okay, so last slide. Sorry, uh if too long, so that's my last slide and then no.

A

It's good because what you're really showcasing here is the complexity of the problem right. So if, if someone's watching this and they're thinking, that's just too much yeah yeah, there is it's so much and and then you'll see in the demo. It takes a lot to look after all this, but we can't leave anything out because if we do there's your Gap there's your breach. There's your problem! So I think this is important that yeah it. We take the time to understand.

A

What's going on here, so yeah I mean that's just my opinion, but keep going yeah.

B

I think you're right always just I'll just quickly add a couple of things.

A

B

You know it is definitely too much, but you know going through. You know everything that praka's been saying. It also assures everyone that you know security focus is in each step right. You know how he discussed. You know we look at the code and if there is a vulnerability in any of the you know, library or dependency, it's you know it pops up right there you know during the build it is a tested. There is a log, there's a ledger: it's a DPG sign. You know in in monitoring phase as well.

B

You know whether it's running you know what we say it's running is there. You know you can see it. So you know it gives everyone that assurance that, yes, you know security is looked at right from right from the beginning right, yeah, yeah,.

A

Well said: yes: okay,.

C

So the last one uh we have this layered security uh throughout the stack as uh Dave was just discussing. It's not just about deployment build is the layered security throughout so end-to-end security from code to build to monitor. We have integrated application, Security checks, our CI CD workflows. Our security focused right from the Inception to deployment.

C

We have a detailed oversight, whether it's a library, runtime containers, entire network, any elements we want, we can watch. We are looking for the security uh in there and open source security software secures profiles. We stay informed about all the open source software risk profiles to ensure that we are always a step ahead ah and we also have the flexibility. Our Solutions are versatile and whether it's a physical setup, whether you are running on a virtual environment, you are running in a hybrid you're running on a cloud you're running on any Cloud.

C

We are just you can bring all these the solutions which channel will be showcasing anywhere. Okay, Let's. uh The last slide. I have before I hand over to title when Tyrell is waiting and his his fingers are like okay, I'm ready. Okay. So we have this high level architecture. So what we have done? We are it's it's it's a fixtures application which is called cool store, which we are deploying. uh We are having a hub cluster where uh all the security components which I was talking about.

C

So we are having red hat Quail, which is a secure uh registry. Where we want to store our images we have, or it had openshift pipelines for our CI. We are using uh openshift data foundations as the back end for for storage, for a hat Quay for sonar Cube, which is a open source Tool uh which we are using for scanning the code. uh Smith any code smells are there so which title will you showcasing? We are using that we are using hashico to save any Secrets any sensitive information.

C

Then we have two clusters Devon production clusters, where our application will be deployed and uh both the Clusters. We are running Argo, CD or it had openshift guitars and that particular application, which is Globex app, uh is uh using Kafka for streaming. So so that is the one which we are not focused for now. We are just focused on the security. Now, let's bring Tyrell in and he'll showcase the soft secure supply chain, Pipelines.

A

Yeah, no, it's a great lead-in because so I'll pull that it's a it's a great lead-in, because now we have the scene set. We know the problem we have the scene set. Now we have to figure out. How does this thing really work and we got we got the person here to do. We got the team to show us how to do it, but yeah Terrell. Why don't we hand over to you and- and you know, let's see it cool thanks, always sure there.

D

We go right so uh for our supply chain software supply chain we're using openshot pipelines. It's a very uh look at it looks very complex, but it's it's very simple in terms of what it does right.

D

um We have an as per car mentioned, we've been, we've got an application called the cool store application, which is a an application that is made up of other small applications. So it is a micro service architecture. Now this pipeline specifically, is going to build the front end of our application for us right. So it's an angular, uh it's an angular source code which is built um and would build them, and a container image is created for that and then deployed to openshift Cluster right.

D

So just to give you an idea how that application looks it's it's quite simple. um It's a front end of a cool store. You can log in and here and you could add stuff to your card and stuff. So it's pretty cool Nifty stuff, just to showcase our. You know our environment in our our stuff that we're going to show you today, all right, so um this uh is obviously built-in openshift pipeline. So this is, if you can see my screen, this is the actual pipeline.

D

That's going to build that container image for us right, um we're using a concept called pipeline as code, so this the pipeline resources are actually stored in a git repository right. So we could take this uh this this code and Department to any openshift cluster right uh and have it uh execute on that cluster right. So we just pass in specific uh parameters into that pipeline and it would build your source code right for this one.

D

Specifically, uh we passed in the Globex UI uh repository first of all, right uh so with any um any source code change or anything anytime, a developer pushes source code into the repository. uh It is actually it will kick off our pipeline for us right. So there's a Web book that is triggered from any push to the repository right. So uh if you've got a list of develop developers, what we we did in this example was that we created a gpg key for each of those developers right.

D

So that means that we actually trust those developers. So we added the public key and I'll get repository for it. So any part of uh so when you're going to build code, we want to make sure that we're building code that's been checked in by someone we trusted right and not by anyone that we don't know, okay, uh so just jumping to the pipeline. So the first step we're going to do is as soon as a developer checks in a change.

D

um It actually fetches the source code for us, so the first step is actually pulling the source code into our workspace. So the pipeline can actually work on it right. The next two steps are run in parallel, which is the scan source and the verify source. So prakar mentioned uh an open source tool called uh sonar. Cube uh sonar cube is actually called in this template right.

D

So it's pointing to our source code and says, uh run a scan on the source code and tell me if there's any bugs or any code smells that are um you know existing in this code and you know and print them on the screen and what it also does is actually creates a report for you on the sonar Cube front end, so you could log into Sony, Cube and you'll. Have a project called Globex UI creating for you and you'll have a list of all the critical medium high low.

D

You know uh bugs or issues that it finds right and all those things are configurable, but for our demo we just use the default and, as you can see, there's not many critical is actually there's no criticals. So that's why our pipeline is successful right at this step. If.

A

So otherwise it would break it there. It would tell you you've hit a critical.

D

Yeah so I daily. What will happen is that this will say failed. uh It'll say refer to you to Sony Cube, you log in you'll see what's going on and then your developer will go back to the source code and correct it. Okay, um at the same time, um what we have is a verify source. So remember, I told you before that when a developer checks in a change, uh we know we trust that guy, because we've actually pulled the gpg public key. That is, that we've generated for him into our git repository right.

D

So this step here is this: basically um ensuring that the commit that was made by that developer uh was signed by our gpg key. So we've got a public key, that's embedded in our pipeline. That says that checked against the commit and says: okay cool. uh The committee is verified because my public event is verified against that government. Right, as you see here, it says good signature and we know the person that uh was dead using so we're pretty happy with that. There's other few warnings.

D

You know, because you know- that's not necessary at this time, but you could get rid of those, but the most important part of it was that we verified the signature right.

D

um If, for example, someone checked it in that, you know is not who we trust. That step will fail. They will stop you right there and says: I can't build a code I, don't trust you, you know, um maybe go and speak to your boss to get your key all right. So once we do those two steps, the next step is building our uh our image right.

D

uh At this point, it's going to point to our source code that we've cloned and start using um a builder to actually build the contain image for us right um once that image is built right. We've enabled techton chains on this cluster, so Techni is going to look at that image and saying: okay, cool I know you built a contain image because, uh with Builder there's two specific variables or results that are set um from that task right. It's called an image digest and an image URL right, so checked on chains.

D

Users, um if you actually look at it here, I'll go a little quickly. um If you look at the build sign image, there's actually two task: Run results that are created here right, uh checked on chains will look at these and say: okay, cool! Your task is actually speed out this information. For me, that means that I need to go and sign this image.

D

Okay, so techno chains will use um the cosine, so we've actually enabled cosine to be the signing Authority for attacked on chains in our demo right. So techno changes will look into the openshift pipelines namespace and look for a secret called signing Secrets right and these uh the signing secret has a cosine key. So it's used to actually sign our our image right along with the password right and then it also generates a public key for us so that we can actually verify that image.

D

So, if you see later on in the pipeline, uh we use the publicly in ACS to verify or see our signature on our images. Okay, so let me just go back to the pipeline so once this step is done, so we've actually built our image and we've push it to query at this point.

D

If you look at queer query here, this image was actually pushed to query and at the end of the image tag you'll see this image has been signed by cosine right, that's great yeah, so another thing that's important is that uh all the security issues have passed right, so we've fixed all the problems that we had as a developer.

D

It was 14 High issues and now we've actually fixed it in our pipeline uh before we execute our pipeline or pipeline, build our image and we all good and noticed it right all right so going back um once our image is built. Now we in this we need. We need to run ACS against these. This image right. So uh there's three checks that are run in Peril right, so two of them are ACS.

D

uh One is an image check, so that's going to check our image signatures they're, going to check if there's any uh issues with with the image we built right uh and the other one is the ACs image scan. Right image scan is going to check your image to say you know. Are you using something? That's not trusted. So is there something that, using from docker.io that we don't? As you know, Red Hat trust right?

D

It's going to flag all those issues for you at this step and you could actually flag them as high uh in critical. If there's any, for example, you want to you, don't want to use any images from docker.io. For example, you could set up a policy in ACS to say if there's an image that contains those layers in there uh we'll flag it as a critical issue and break your partner. Stop your pipeline understanding.

A

So if the built their own image, they could be using layers from places they maybe shouldn't be or don't trust, and so that's, where we're going to get into that. We can actually start to look at those layers and say: hey wait a minute. This is fine, but this is it right. Yeah, yeah.

D

So ACS will actually print that out for you on the screen and saying you know, I, don't think you should be using this image or this image from docket IO is very insecure. It's you know higher change it. You know um all right. So those are the two ACS checks we do and the final one is what prakar mentioned before is the s-bomb generation right.

D

This is actually going to go and uh scan your node.js application and look at all the dependencies that are being used in that application right and it's going to split out um a file called uh an esbom right. So this is an XML representation of your Expo right.

D

uh It doesn't make any sense to you know just looking at it right, but what we added recently in the chain, so it's got stuff like you know which Library using the dependency is where it's coming from uh it's it's just basically like a library of your library right.

D

um So what we did in the latest release is that you know people ask you know: what do you do with this s form? You know what can we do with it later on? You know. um So what we did was we added a tool called gripe which is actually going to scan your S4 right. It's going to scan it and it's going to so once we build the S form at this step we actually scan the as well and it'll.

D

Tell you all the libraries that you've been using and the severity of those uh libraries. You know in case, for example, right loaded utils uh has a critical severity. So you know it's flagging you that you're using this Library uh 2.02, which is as a critical security issue, and it can be fixed by just installing 2.03 so for this demo.

D

We didn't want to break the pipeline for this, but you could actually break the pipeline or stop the pipeline if you've got a critical or a high issue, and we should force your developer back to the development stage to actually pick the right, correct libraries to go and uh build your application and.

A

It's so easy to read so you're saying this was just recently added because I I mean I've been talking to you guys about s-bombs and besides sounding really cool, you know check the s-bomb sounds like a super spy thing, but like this is great. Like I mean I can deal with that right. I can see that I can understand that yeah.

D

It's cool so before we didn't have it so a lot of people ask you know what do we do with this stuff? You know it yeah so, but now it's just an example of what you can do, so you can actually build rules around this in your pipeline to fail your pipeline or to send out alerts to you know your. uh You know your Ops Team to say these are the issues you know flag them when you're. You know in your your libraries that you are pulling from and make sure that you don't use them.

C

Was just to maintain the compliance, but now not just compliance. You can actually scan that and find out the vulnerabilities.

B

Make it sound really easy, but it's such a critical step. You know this s-bomb, you know it. Could you know, save your bacon at the end of the day. It's it's so important. They just see that one critical severity uh being missed and you know you you get into all those. You know the numbers that you spoke about, August immediately right that one. You know one small, you know Miss and you know you could you know talking Millions right.

C

The one which I was talking about so s-bomb is one of the parts of that uh compliance to maintain that compliance. They have to generate as poem for every soft piece.

A

Of software they are going to put in U.S, federal, right and and other nations will follow. You know or or need the same thing. Europe will do the same. Australia do the same, see superheroes, it tells you. This is what we need.

D

Cool all right so once we've generated our s form and got the report out, uh we in a position to safely deploy this to a Dev environment. You know we want to test our image now right. So this next step is that what happens now? We've got um our CD process sitting uh outside of this, so we don't manually push our image uh from the CI into an environment. We've actually updated a repository uh using which I'll go CD points to, uh and basically we pushed, we updated.

D

The repository with the new image tag right, so August CD will pick this change up now and notice that our repository has been updated and deployed this to our development environment right. So, if you look at this image here, um we've added this tag in our pipeline right. So you say that you know. uh Oh sorry, this is what the developer has actually done in the source code. He's added this tag uh to the front end, and now it's built image and it's deployed to a Dev environment.

D

If you look at what product looks, it doesn't have that right. So this is the product instance and it doesn't have that tag right all right. So the important thing, though, is I've shown you the the bull stage right. So another thing that ACS can do is that it can prevent your uh your images or deployments uh into your your environment uh if your image isn't signed, so what it does is it will check the image.

D

That's in your deployment scan it to determine if there's a signature on it and if the signature doesn't is not what it expects. It'll stop you stop that from happening right. So that's pretty cool. If you've got developers that are just deploying you know, whatever they want onto your environment without knowing right, so you can force your developers to make sure that they use sign images that are trusted by organization uh to your environment. Right.

A

Okay and it makes sense because developers need to get their job done, and maybe we don't force, we guide them to use the correct, the correct images and, and then people learn right because I think that's a great thing is we can actually start to educate the work. You know that the development development team as to why they need to do this, because so here they are seeing it going. Oh I didn't realize yeah, correct.

D

So what I did was for this demo is that I built another pipeline called uh the Globex UI unsigned pipeline. um So this here is it's pretty much a very uh simple version of the previous one. All it does is pull the source code, builds the image, pushes the image to Dev and does an ACS check right.

D

So what I did one in this specific pipeline is that I intentionally left out a technon chain signing so I forced the image uh to be built without being signed right and then I deployed it to Dev at this step right and then what I did after was after I deployed it I'm scanning the image that I deployed uh using ACS I'm doing the same check. I did in the previous step, and then it says here: look I can't uh build this pipeline.

D

It's not successful, because the policy that AC has triggered says that that image is inside, so it'll break your pipeline. If your image is inside, so that's one step right. uh The other thing that I've included was the um trusted signature policy on deployment right. So if you look at this diploma here is actually deploying an unsigned image. So when I push that image to Dev, this is the actual deployment.

D

That was that's using that image right and if you look at the events, uh ACS admission controller is saying: I can't deploy this now, because uh The Trusted signature policy is prevented.

D

Okay, so I can stop my my uh you know: I can prevent anyone from the front unsigned or untrusted images to my open, driven brand.

A

All right, that's cool, so that ACS stopped that I'm viewing the deployment on in my openshift console it's all working together. Oh that's, really: cool yeah! That's pretty nifty what you can do. The policy! That's really cool yeah! All.

D

Right so we started out with Dev I just showed you what ACS can do in terms of the you know, signature verification. uh Let me go back to our original pipeline, uh so we're here now, so we so once we deploy to Dev um what I added is a very cool feature from ACS, so there's another feature that ACS can do is that basically it's can scan your deployment file. So your actual openshift deployment file and can pick up any issues that may arise right.

D

For example, in this example, I've intentionally left out the request and limits for your deployment of for your container right, so I've set it as a medium severity and just for alerting purposes. It's telling it's full! It's telling you that! Oh, you know what, uh when you created the the deployment file, you forgot to put in a CPU limit and CPU request right, uh so ACS can give you that flexibility. It can tell you not. You know, you're, not adhering to your development.

D

uh You know, structure, you know, rules or you know whatever the the Ops Team have provided. You know you need to provide these specific values in your deployment. It'll, stop your your pipeline from the point from building at this step as well. So you could add. That's a pretty cool thing that we could do in just to enable your developers to to build things correctly intelligently.

A

Yeah, that's that's neat. It's really neat all.

D

Right so, uh along with the ACs deploy check, we've got um I've just added something uh from 60. Has a tool called Rico? uh What happens when you sign your images with with cosine it pushes metadata about that signature to a third-party service called Rico right? So it tells you about. You know the time it was signed who signed it. uh You know the information around the signature, uh you know.

D

Basically what you could do with this is that if someone is using your want to use your image and doesn't have your public key or doesn't have access to, um uh you know Vital Information, for example, you have to give your public key out to someone for them to verify your image. If they don't want to do it, they could actually query the record library and ensure that this image comes from a trusted source.

D

So that's just an additional thing that we added we've got ACS to do that for us in our example, but uh this is something that you know that's available by the six store environment, open source which allows you to actually look at the signature metadata.

C

Yeah, this is something which is making easier for the customers who wants to develop the code or container images for open source, softwares and I'm going to push it to the public repositories. It's for them. It is very much easier like they don't have to get the public key. They can go to wrecker, find in The Ledger. They will know that exactly this particular image was signed by XYZ company yeah.

A

So that's great because it really Fosters that sharing and and builds we keep saying it but builds the security step in it's not. Oh now, I mean you'll still check things, but it gives them that rex that extra assurance.

D

So this step is just you know, showing you uh I'm querying the the record store for us at this step, and it shows you uh metadata about your signature. So it shows you know uh the the image registry repository that you're pushing to the digest of the image. um You know the timestamp. You know the tech, the task run, the actual task run uh ID that was used to create that image. So it's it's a whole complex. uh You know not actually complex.

D

It's a very thorough uh view of of you know of your signature, which people can look at and with. If they want to trust it, they could use it. You know, so it's just something for them to you know, settle their mind, basically right.

D

So once those two steps are done, we're ready now to actually prepare our deployment for production right uh now. Production is obviously a very secure environment. You know we don't want to do the same thing with Dev is actually push it directly to our branch and have uh it available in a developing environment. uh What we do is we've. We create a pull request, a branch and a pull request on that Branch. So that basically means that uh someone with the correct privileges uh will log in here.

D

uh You can go to the pull request uh that we created here and then commit this right, and then, basically, this is merged into a branch, but Argo CD will pick up the change now that we've we've merged into a new branch and deploy the image to production right, yeah, so I know we're running out of time, but uh what we could do is, uh if you want to I, haven't, showed you how we kick off the pipeline right. So um what we can do now is I'll.

D

Just give you an example of how this is done. So just for example, a developer wants to make a change I'm just going to edit the readme file for uh for ease of use, so I'm going to go and test I'm going to save this I'm going to push. Now to my um this is yep. I want to push this to my Repository.

D

And then now it's actually pushed to my git repository right now. If I look at my pipeline, this should automatically kick off a new pipeline yeah, pretty.

B

D

So that's pretty integrated with your your developer, your git repository and your uh your your CI CD pipeline yeah. One.

C

Of one of the things which we want to bring in here is we don't want to uh overload the developers the developers are there that they can do best. They develop their code rest of the things we will automate it we'll take care of it. Openshift has all those tools to do that, for them worry about this.

A

And that's what I was thinking is. How do you remember all this right as a developer? Imagine the context. Switching and and I mean even I I think in the operation space as well. There's there's room for this, but just even just with the developers who are essentially writing. You know they're very in that mode.

A

Imagine the number of mistakes they're making if they're dropping out they're coming back in and they're dropping out and they're coming back in to take that away from them, but to have that level of checks and balances and complexity behind it is just brilliant, isn't it right because I mean and as a developer, which I'm not but I would feel much better I mean.

A

Imagine imagine you know if you had pipelines for like raising a baby right and you had all these checks and balances in there as you went through their life and it was like, oh well, when they you know they start eating solids. You have to do this and it just gets done for you and you could just focus on parenting. Wouldn't that be good, but that's a weird example that we probably don't want to investigate too deeply, but because that goes into Ai and robots and scary things like that.

A

But I think this really calls out the importance of yeah, of letting them do their work and and do it well um and- and you know excel at it, so I love seeing all these pieces yeah it's pretty cool yeah.

A

It certainly is, and so I mean yeah. It's it's it's really. It's really neat to see, and this is all coming as part of um openshift and and that you.

C

Know so all most of the tools which we have shown you is a part of openshift platform plus so that ACS would have Quay uh openshift platform, openshift pipelines, Argo CD is openshift, get off so they're all part of uh openshift uh platform, plus SQ, and uh some of the tools like sift and drive red hat is supporting them the open source tools. We are trying to get them as part of our offering, which is uh trusted application pipelines. So that's the idea uh some of the tools like solar Cube.

C

They have their own way of support. So, if somebody wants like so not you with just an example for us to show you, okay, gripe is just one of the examples like maybe in future. Acs can do that or something else can do that. So.

A

Right and so it's not only like it's pluggable right, so we can, if you're, working if you're a an existing business and you like, but our developers do know sonar Cube. So, let's use it yeah go ahead. Of course. The point is to make everything better together to make it all kind of, because why? Because we need developers to make good code so that good code gets out there and customers get what they need safely and I.

A

Think that's what we're really showing here is you know whether it's where it doesn't matter where it's running, we want to make sure it all works together, and that's the other thing that you guys called out that I, love and always love to talk about is that this runs the same. Wherever you want to do it, you can do this on premises. You can do this on the cloud. You can do this in all these different places and it and it runs the same.

A

We get that consistency and portability and so I think that's really neat I.

B

Think it's a win-win, for you know, developers and consumers. You know, there's a One-Stop shop, you know everything comes into one place. Developers can write, you know, quality code, you know, hassle-free not worry about all these checks and balances, and you know at the end of someone using this, you know is rest occurred. You know, everything's been taken care of, and you know you have all the logs and the ledgers to you know check if you want to they're really predicted yeah.

A

B

Really like that,.

A

A win-win for developers and consumers I think that's a nice statement that kind of reminds you that we're actually all linked and when we're all winning is great yeah that no that's really good. So yeah do you have any more to show here Tyrell on the demo or we've I mean I, know it's still running, but yeah we've basically seen the bits and.

D

Pieces yeah, that's that's my.

A

Subject, this is great cool. Let me just well then, if we're getting lots of rain here in Sydney, it's kind of exciting that, finally, to get some rain after a little while well, there you go, we've had a great look at the story behind trusted pipelines and the understanding of security at every step kind of built in it's like you know something, no matter where you go, you have it um and something we love.

A

I I think you've basically helped with all those fears about the numbers and stuff I I, don't know I'm, sorry, guys, I, don't know if you've solved my problem with the Broncos. uh They lost, like I, said by 50 points, and um it's going to be a long season, but uh I feel a lot better that when I'm accessing their app or playing fantasy football or whatever I'm doing it's probably going to be done safely and I'm not going to have my details compromised, so you know we take the wins where we can find them.

A

um So I think that brings us to the end today. One thing I do want to remind everyone who watches the stream and re-watches right now we have red hat Summit connect going on in aipac, and this is the team behind the demos that you'll see.

A

If you attend one of these events, they provide the environments, the support um and a lot of the knowledge to back our red hat teams and make sure that as customers are seeing our software and our products as they really are, so you know try to come along to one of those they're all over Apec, there's nine or ten of them. um Some are done. Some are still happening and, of course, as well. uh You know we got tons of other shows that are streaming.

A

I know Europe's waking up now, so you know, there's a show that runs in Europe check these out and enjoy them um and and beyond that I'm so privileged to have had these guests it's great to have everyone back uh I can throw it around the room one last time to to wrap it up. If you like, um I, don't know if you have anything further to add, but I just really appreciate you being here. Well,.

C

Good did this development secure interest yeah.

A

Tyrell you know, I can see you love.

C

A

I mean I I feel, like you know, I I should hire you as a developer. It's.

D

Terrific yeah, it's pretty nifty stuff and you know uh the stuff that red has been bringing out. It's really cool. uh You know it helps you a lot to minimize. uh You know configuring everything comes out of the box, you know, so it's pretty pretty easy.

A

And we'll share links, I'll put some links in the description and stuff and finally Deb it's great to have you back, um and uh you know it's good to be back we're gonna, look at some new shows, as we finish off the year. I think you'll see these guys back uh uh soon, because we love seeing these demos and we love really just hearing it. You know at the Grassroots level, so that's really cool, but Dev I, don't know if you have anything else to add.

B

Thanks, Augustine and I have no doubt you know, prakara and Tyler would do justice um love the show loved. You know what the content that they shared I think it's. You know you could share this and whoever you guys watch this should share it with you know other developers as well. This is some really good content here, um thanks again for being on the show.

A

Yeah absolutely well said, as always: okay, folks, we'll wrap this up um and we'll catch you next time on the copy hour thanks, everyone see you later bye.

A