Red Hat OpenShift The Level Up Hour | Red Hat Livestreaming, 9 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Level Up Hour (E37): Container Catalog Health Index

Description

What does the "Health Index" of a container in the Container Catalog really mean? Should I only use things with "A" ratings? How stale are the ratings? In this episode, Red Hat Principal Product Manager Brian Cook joins the show to discuss the Red Hat Container Health Index, how it helps identify security risks with Linux containers, and what it means for your containers. Tune in to delve into what we should be looking for when choosing container images!

More about the Container Health Index:
https://access.redhat.com/articles/2803031

Follow us:
Langdon White https://twitter.com/1angdon
Chris Short https://twitter.com/ChrisShort
Brian Cook https://twitter.com/brianwcook

What is the Level Up Hour?
Every Wednesday at 9am ET we stream live demos on how Red Hat Enterprise Linux (RHEL) users and admins can benefit their organizations and improve their careers by learning how to use containers, Kubernetes, and Red Hat OpenShift.
Learn more at:
https://red.ht/leveluphour
http://red.ht/3ayHlHI

How to participate in the The Level Up Hour:
Watch on YouTube every Wednesday at 9amET
Chat with us on Discord https://discord.gg/5VMVGJt
Check out our GitHub repo https://github.com/level-up-hour/

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
https://www.openshift.com/
https://github.com/openshift/

#RedHat #OpenShift #tluh

A

A

A

A

A

A

A

A

A

A

A

Hello, everybody uh today we do not have chris short, so uh as a result, we're having all kinds of challenges, as you might have noticed, but today I have with us some pretty special guests, somebody's running the twitch channel with the live sound.

A

But today uh andrew uh do you want to introduce yourself as my uh as my guest co-host.

B

Yeah, although you can't blame issues on the lack of chris short, that's that's not! That's not fair!.

A

Yeah well you'll see in the slides how I further so not to worry. um Yes,.

B

So hello, everyone- I am andrew sullivan technical marketing manager uh peer of langdon. I also host the show that is happening today at 11am, uh so with chris being out today, taking a well-needed day to rest and do his thing uh lane and I are joining each other's shows as as co-hosts, so uh look forward to seeing um langdon on my show later and being a participant here, um although I admit I am not a developer by any stretch of anyone's imagination, so I'm I'm hoping to learn an awful lot here.

A

Nice all right so uh to make the show even more fun uh and to taunt him further. um We have invited a special guest uh named brian cook um and we will we'll talk about the slides in a minute and talk about what the show's about uh in a minute. um But uh brian you want to introduce yourself, tell us what you're cause I. I can never remember anyone's titles at red hat anymore because, like every you know like three months, they change. So you know I always like people to introduce themselves.

C

Sure, hey, I'm brian cook, I'm a product manager here at red hat, I work on an internal product that we call the container factory. It's actually kind of a big coalition of people inside red hat, who work for many different teams and work to make all the machinery that produces all of our containers and operators and all the stuff that has to go along with building and shipping those work, and it's actually quite a large group of people.

C

It's an amazing group of people um and the fact that sort of it's a a cooperative and not a forced function. That makes it a kind of an interesting thing here.

A

Nice, um and so why, and so some of your work leaks into the outside is basically what it goes on, I'm sure it leaks what into the outside versus just being an internal project. Oh.

C

Yeah yeah, we have a lot of code that um we push outside and we're actually making even more efforts as we change things so yeah. A lot of our efforts become open source kind of upstreamy things and some of them get included into products or things like that.

A

I gotcha uh cool, uh so why don't we I'll do the my ugly ugly slides? um It's still bothering me that I can't seem to get into the twitch live stream to watch the chat, um but I think I finally figured it out so now it should be better, uh but I get no history on chat. So if we missed anything, uh please let us know. um So. Let's see technology, try not to defeat me for a minute uh and oops.

A

It's defeating me here we go.

A

Oh, hey: guess what we're gonna share on weyland and uh things are going to be not as.

B

Useful first time sharing on a new laptop.

A

Yeah exactly my uh my laptop wouldn't boot this morning, so that was fun. um I suspect it ran out of disk because I was doing an upgrade over the weekend.

A

And none of these desktops that it's presenting me look like the desktop that I want to share.

A

This is super weird, oh, let me just uh try this and try this all right. We're really uh firing on all cylinders today,.

B

It's an ominous start to a wednesday morning exactly.

A

All right, so you should be able to see the slides now, even though the aspect ratio and the sizing and all that is probably off um but long story short uh here- is the level up hour. Normally we talk about containers and so containers we can do. uh Apparently uh you know streaming booting. All those things are are out of my wheelhouse, so hey lincoln.

B

It's sharing your whole desktop, so you might want to maximize that window.

A

Yeah, that's what I was wondering uh all right: it's not too it's very, very big for me now um so uh yeah. uh So this is the level power and today, uh as as talked uh discussed a minute ago, um I'm langdon white and uh you can find me on twitter at uh langdon with a one um and chris is not here today.

A

uh So uh we have crossed him out in in uh in a particularly harsh manner and invite mr sullivan in his stead, uh who you can find on twitter at practicalandrew, so join us there. You can obviously still talk to chris short. He will be back next week um and uh you know all the better for it. I'm sure, and uh you can also find us on our discord uh and there is the link there. um Let's see, can I use the magic twitch, let's see if it works for me.

A

uh No, I have to be logged in um which I am currently not, so uh you will have to type in the link yourself or maybe somebody else can share it if they feel like typing, so yeah all right still trying to calm down from my panic attacks this morning. uh So you can find out more information about the level up uh program in general which, as we talked about uh offers some free training and loc and like cost reduced training as well as licenses to open shift.

A

uh If you want, if you go to that url, you can check it out and today we're going to talk about the container health index, which I find particularly interesting and I'm not sure a lot of people, aware of um as we usually do. uh We have show notes from last time which uh you can find at that link there, and I think that one I actually managed to uh prepare in advance.

A

But I don't know if it's gonna, let me so let me chat it all if I'm not logged in no, it does not hold on. I have to log in um so oh good gracious. Now I have to tf 2fa.

A

uh So, um let's see, but why don't we get started, and I will sort that out and give you links in a second um and I think that's the last slide. We want to talk about for right now, um and this whale and sharing thing is going to be fun for any kind of uh uh browser-based activities, um because.

C

It's not a lot better at dealing with light weyland lately, but.

A

Yeah, I didn't uh you know I actually had. I just uh had been talking to uh some of the desktop team about just that um and uh I was. I was actually really surprised that firefox, wouldn't let me share it, I wonder with chrome. If it would, let me do it, um you know, let me do two different versions or whatever.

A

uh So let's talk about the container health index, which uh why don't you tell us a little bit about what the goal is, uh brian and uh and then we can go kind of from there.

C

Sure so, uh as we all know, well, maybe we don't know I'll just say: containers are immutable right like uh once you get one, it is what it is. uh You can theoretically create a new container from the original one right, but then it's a different container, not the one you started with.

C

So the container itself is immutable and kind of our normal pattern for using them is like get one use it for a while and then um throw it away and get another one from the source not not like uh continue to to patch the one you have right. So we talk about like pets versus farm animals, kind of methodology with containers and the the problem is like. When do you know when to get a new one or like what?

C

What um when you're, particularly if you, if you use our ecosystem catalog to like look for images that you might want to use like?

C

How do you know like what the status of that image is, and the container health index was conceived a few years back as a way um essentially to understand, like the the containers status with regards to available patches? And that's like a critical thing to understand, because actually, um originally the container health index wasn't called that it was called the freshness grade and it was. um It was meant to to convey like this. Isn't you know we're not trying to say uh this container is safe right, it's just like how new is it?

C

How recently has it been updated um and the container health index has grades like sort of like school, except for the end of it gets a little weird like we never had ease.

A

Yeah, I've actually always wondered about that. Why? Why don't we have ease? You know, I don't know, because is it because it collides with excellent that's what I was saying. That was my only theory.

C

I'm not sure I thought maybe it was vowels and I realized there's a vowel, so I don't know, but uh so the deal is like um we're. We're trying to convey the message about the most important missing patches and for us that is, um the line is for important or critical right, so cves have ratings rating system and the top two are critical and important. After that. It goes to moderate most of the things that we rate as moderate are not. You know.

C

Super concerning to be running for a while on a production system, um while important and critical is the ones we think people should be focusing on the most, and so the container health index only deals with the lack of important and critical vulnerabilities.

C

um There's a scale, and I shared a document, if you guys want to put it on the screen. Oh kb, kb article, um it was just painter health index tv, article.

A

Yep sorry um yeah, I got it.

A

Speaking of logins, uh so what I wanted to point out real quick was see if you notice over here on the screen that I'm sharing is that health index right here. That is what we're talking about, um which may not be obvious. um Okay, so, let's make that a little bigger, so people can read it. That's what you're talking about right.

C

A

C

A

Oh, the main one, sorry.

C

A

Yeah, I am really firing also.

A

I also can't seem to get my mouse from uh the one screen to the other. That's why you see it popping up. Oh look at that. Hey! That's, really useful! I've never found this before.

C

So this is on the uh the the public knowledge base, and this explains what the grades mean in detail right and it shows the grades there so grade a no missing patches, essentially like no no missing critical or important cd patches, and then you go down to grade b, and this is when um the factors start affecting the grade and the factors is what kind of patches um critical or important and they affected differently and how old the patch- or you know how many days since that patch was released and that's important right because, like the longer the fix is out there or the longer, the the vulnerability is known.

C

The more people take advantage of it right and so the same vulnerability will get a lower grade. The longer it's been um fixed, so grade, b, grade b, like uh is missing, critical or important security. No missing critical clause older than seven days no missing important flaw um older than 30 days, and then you know you guys can read. But as you go down, it's like uh it's older, older older until you get to grade f, which is like you're missing stuff.

C

That's a year old, right um and- and this is this is uh something you will find in our catalog for the simple fact that, like we have support life cycles and when something goes end of life, we stop rebuilding right. So um right, if you go and look for like some super old uh image in the container catalog like it, it will be created f, it should be graded f. We want you to know, it's graded f, don't use it.

A

Well, is there is there a distinction between you know like a grade f, because it's a great f and a grade f, because uh nobody cares about this anymore like like, or should there be in a sense.

C

In the container, catalog there's also a uh a release category on that page. So if you want, you can flip back over to the the um rail page or I actually linked the ubi 8 page there, which I had looked at earlier, that I know there's a tag on that has a different grade.

A

C

There's there's a release category and it should say generally available if you know you're looking at something that is in its support life cycle.

A

Okay, all right um yeah. Let me uh you said ebi. Oh ubi is here. We have used the ubi on the show many times, ubi.

C

Yeah, I gave you a link in the doc yeah.

A

Trying failing but trying, let's see if this is to the right place, I guess the one right.

C

A

So you should have.

C

A release category of generally available, um oh.

A

Here yeah, I see it now.

C

And then you have your grade as a right, but if you go to the top, where that tag says latest yep and uh pick the tag, that's two months old, it's like 8.4 dash and uh uh 203, and uh and that's which I'll tell you what that is in a minute.

C

Yep gotcha you'll see that that one is not an a it's b right and if you click on the security tab there, you can see why it's a b right, because it's missing that uh that important vulnerability.

A

So but if I need this 8.4 203 plausibly right, I could put a layered container over it and update this cde right.

C

You could, but that latest tag that we were on before.

A

Already has it right right right, but if I needed like if I was more like if I needed like 8.3 or some weirdness like that for some dumb reason, um theoretically, I could overlay it and apply those patches you could. You could do that. Obviously, there's some challenges with that, but you know you you can.

C

The challenges with that which uh the point releases of rel are not supported after the next one is released immediately. So.

A

Oh, that's true right, yeah right, but if you're, if you're playing the catch-up game um yeah, you know, that's uh it it'll bridge the gap, while getting you know being able to defend you against some pretty major stuff. um Although you know nine times out of ten, at least for me right. The reason you can't upgrade is the same reason that is flawed right, like like they're, usually related, and so that's going to be.

A

You know, because you get a little stuck anyway, no matter what you do, um but I got you um cool all right. So what um you know so you know: do you have a lot of people kind of giving feedback on this I mean. Are people? Are you hearing feedback from you know, community from you know, customers or whatever about using this info.

C

Yeah we get, we get feedback, we it's positive and negative right. um The the things to be the negative feedback we get is all focused around um some of the gotchas right, and so like it's important to keep in mind again that grade a doesn't mean we're saying it's safe. It just means it doesn't have any outstanding patches that need to be applied right.

A

C

Depending we do, we have a very good security department who ranks the patches, but, depending on how exactly you are using that container. A moderate, moderate vulnerability, which doesn't factor into the health index at all, could be more important to your use case than we think it would generally be right, um we're, I would say, we're pretty conservative on the security rankings, but it's not impossible right.

C

Another thing to keep in mind is that, like there's a lag, that's there is some lag between when a patch becomes, uh let's say like a like. Our vulnerability becomes known about and a patch becomes available right. So this the container health index is about uh you're missing a patch, and so we drop the grade so consider this scenario where uh a critical but not like embargoed right- and I don't know if we need to explain that.

A

Actually, yeah, let's so embargoed right so when when there is a secret patch, basically.

C

These are the worst of the worst right vulnerabilities right. The super worst vulnerabilities uh that that you can think of the ones that make news headlines right um like when those things are discovered. There's you know. uh Typically, somebody will say I found this super egregious vulnerability and it's so bad. We're gonna we're gonna, keep it a secret until we actually release the fix, and so that's called an embargo.

A

And it, but it's not it's only a secret from the general public, usually the security teams of a bunch of like the linuxes. Even the windows, people, like whatever are often communicating about it,.

C

That's necessary right, necessary essentially for global information security, because the same the same patch or the same cve will will typically affect many different operating systems. Right it can, it could affect, uh it could affect red hat, you know rel and fedora, it could affect ubuntu, it could affect windows, I mean it could affect tons of stuff and so yeah yeah. All these companies have to work together to prepare their patches and then then there's an agreed upon release date. Where for the vulnerability to disclose.

B

So brian we've got a question uh from playing risky uh about the security reports, uh sure so more or less. The question is where, where does the? Where does that information come from the the various uh security alerts that it's testing against, um so the specific or the precise question is? uh Is the security report integrated with red hat errata and claire, or where does that info come from.

C

It is, it is integrated with red hat errata and specifically the oval streams, and it is using claire to produce the list of vulnerabilities. Yes, I um put some resources in the dock late earlier, because I assumed this question would come up and we would- and we would probably trend in this direction for anybody who's going. What are you talking about?

C

uh Oval is a common um syntax for describing vulnerabilities and fixes to those vulnerabilities, and I have a link to the oval kb article for you guys and also a link to the rel eight oval files. So you can see what those things look like.

A

Yeah, I'm not sure we want to look at a bz2 necessarily, but you know now you can send it and send it to them. um Oh yeah, that's a good idea I'll, throw up the chat um unless andrew you already beat me to it, but now I'm scrolled back! Oh look at that! uh Oh you got the article too. So.

A

Yeah so- and this is not like like oval- is not a red hat thing right, it's a.

C

Yeah oval is an industry standard language right or describing cves, and the available fixes to cves from different places.

C

Red hat provides oval feeds for its own products and for open source, open source distributions like ubi, and there are other oval feeds out there. If you go and like uh search around on the nist site, you'll find oval feeds for lots of other software as well.

A

Yeah it's actually. I was just kind of going back to the embargo thing too, um like that's one of the weirdest things I think about having now you know kind of working in an operating system company. You know- which I am you know now and have been for quite some time, but before that I never had right. Is that, like I don't know about the two of you, but I often am aware that we have x y z, like some number of embargoed patches coming.

A

I don't know when they're coming, I don't know what they actually are. I don't even usually know like how egregious they are. I just know in the ether somewhere there's an embargo attached. This actually happened a couple of weeks ago, like this. That's.

C

And that's by design right like right right, it is literally need to know even inside red hat. um I I am responsible and part for most of the stuff that ships our containers and I am not informed when we have an embargo.

A

Right right: well, I think that's what like, I guess you know like kind of looking at it from the outside. You know I kind of always assumed everybody at red hat knows and they just keep it secret. You know, but in fact yeah like next to no one knows um you know it's basically in.

C

Fact on the development team on the development teams, it's usually only developed only the developer. That is writing. The patch is the only person who knows what the actual issue is.

A

Right and not even their manager, a lot of the time which I think is particularly interesting. um You know unless there's some, you know a lot of times. The the managers at red hat, particularly in engineering, tend to be like lead engineers as well. So there's a little bit of a you know can be a little bit of a misnomer in that they they will often be consulting with the developer themselves about the patch um but yeah, it's uh I I don't know.

A

I find it really interesting um and uh you know, but they're and they're, not even all that bad. Like I, the one uh that was relatively recent, I I remember I was a little surprised that it's uh cve rating so like there's ratings from uh whatever one to ten, I guess or zero to ten, um and this was and ten being the worst, and this was like an eight and a half.

C

A

Yet it was embargoed um which you know, that's that seemed at least to me a little low for an embargo level uh cve. So it's a pretty rare event to do with embargo. But you know yeah. If.

C

It's usually a process. uh We only process, maybe like two to four a year right. You.

A

C

Generally, just a couple a year.

A

Right, ssl guys are uh doing their thing.

C

But actually why I went down that rabbit hole, as I was saying, imagine imagine a vulnerability that is not embargoed like a critical vulnerability. That's not embargoed! Okay,.

A

C

So um the world knows about the vulnerability and there could be uh a week or a couple weeks you know depends. But there's there's some time gap between when the world knows about the vulnerability and when we ship an rpm that fixes the vulnerability right. Okay and then there's some other time built between when we ship the rpm and when the new container is respond. That includes that rpm.

C

So in the first time, gap between when the vulnerability is disclosed and when we shift the patch the container grade does not change, because there is no available patch.

A

C

So, even though there's a little grade yeah the grade reflects the status of its outstanding patches. There is no patch, you know it's there, but there's no patch. So, even though like it might be an important, critical vulnerability. That container will still say a then in the second time gap um there is a patch because an rpm has been shipped that that comes with an rhsa. So that is the errata that the questioner was asking about before right. So we have shipped an errata, a red hat security advisory.

C

Now a patch exists and that rhsa will be reflected in the oval feeds. Claire would be triggered by that errata shipping to update the security tab. It's a metadata behind the scenes, but what people see is the manifestation of that is that security tab in the catalog you're going to now see okay, this thing is there and the grade would be impacted.

B

So I think, that's a that's an important distinction to make right of it's not doing vulnerability scanning it's doing patch checking.

C

Correct what claire is currently doing is looking for missing patches, it's because it's using the uh the oval feeds that talk about what patches are available for installed. Containers.

A

Right, but in the same way that um you know if I configure my httpd uh container to be writable, uh you know or to have the you know the files writeable uh from the world, um it's gonna be pretty insecure, but there's really no like there's no scan for that per se. Unless you get into like kind of straight yeah,.

C

Security scanning yeah when you get into runtime scanning and there are lots of companies that provide stuff like that. um But that's that's not what this is right. This is a static, we'll call it static, vulnerability assessments, and that means like we don't know anything about how you're going to run it. We're just looking at the container itself.

B

So brian, what what triggers a rescan of an existing container? Is it the release of new erada? Is it just a periodic? It happens every you know: 1 2, 6, 12 hours.

C

Yeah so, interestingly, the way claire works and one of the things that's fantastic about it. We don't have to rescan the containers ever.

C

What claire does is there is a database and when we produce a container, it's it is scanned by clear and essentially what claire is doing is creating an inventory of what's in the container, and it puts it in this um postgres database that kind of kind of behaves like a graph.

C

Then, when we ship an errata to follow langdon's example um say we ship in a rod against http d2 um claire, gets notified by our message, bus that we've shipped in errata, and it gets information about like what what vulnerability was shipped from oval right and basically we just kick it and say like there's new stuff, just to be a little more proactive. It would do it on its own anyway, but we can kind of push it right, and so it will go, get the updated oval feeds and inside that oval feed.

C

There will be information about the errata that was shipped for the package. Then it can look inside its database and see all the containers that's ever shipped with that package and the vulnerable versions that were patched and it can immediately link the new erada to those affected containers so that that takes care of claire sort of knowing about the the vulnerability.

C

We have an extra step wrapped into earth in order to make the catalog and the container health index work, which is, um we have a separate database that powers this ecosystem catalog view that we're looking at and when claire updates that information.

C

It takes a subset kind of a summary subset of that and it updates the catalog as well. So that is what finally sort of updates that security tab view and the container health index grade.

A

Well, so kind of in a related way um do the containers in the container catalog ever get updated aside from a security patch or whatever um they do. They do.

C

uh Well, they get updated because we push new features or release new versions of software.

A

Right so like 8.3 to 8.4, for example, sure.

C

A

I meant more, like um you know, is there or or like okay going back to, or maybe that is kind of the related thing so going to you know whatever this, um you know these patches here um you know 8.4 199, whatever uh are these always security driven or are they feature driven as well? Although I guess.

C

There shouldn't.

A

Be any features.

C

um uh Okay, so there's a do, you see those tags that say, let me you're just really small. Let me pull it up, uh so the tag we were looking at that says like.

C

8.4-203.1622 6601 201, whatever that crazy thing is right, yeah, the part after that, um the end of that string is an epoch right like you next time, uh and whenever you see those tags in the container catalog, you can immediately know that that was triggered by us shipping a cve, because we have an internal automation process called freshmaker and it is triggered by us shipping cves being special.

A

C

Yep yeah and it does a bunch of stuff now, but uh freshmaker essentially, will be triggered by us shipping and cve and queue up all these rebuilds and process them automatically to build new containers that include the newly shipped patches um we do have. We do have tags like you'll, see, 8.4-199 and 8.4-203 right like that is a that is an example of what you're asking about that was. That was a tag cut by the rel engineering team and released for some reason. I can't I can't tell you exactly what the reason was.

C

um It probably does include some like lower level. You know, like maybe some moderate security patches or something like that. uh Maybe.

A

Like api improvement or something yeah.

C

Right, you know, you remember the erotic categories. We have bugs security vulnerabilities and enhancements right, um and so the fresh fresh maker only responds to uh security vulnerabilities right.

A

Okay and that's kind of where, but.

C

Relent teams on a release cadence, where they'll they'll hit their release. Cadence and they'll cut a new version of like a new container with all the updated rel stuff in it. That's not security related as well. So, yes, you do get those um inside of like point releases and you can tell the difference by looking for those epoch stamps on the tag and.

B

Just just to be clear, once a container is pushed, it's no longer changed right so effectively over time. It slowly marches downwards in grades as more releases or more errata is released against it. So if today I pick a specific tag and it's an a next week, there might be errata that degrades it to a b and then the following week, a c and the following week, at d and so on and so forth.

A

Yeah, so I just I just plopped over to 8.0 right and it has a nice nice fat f right there yeah and that's exactly what it should have right right. So, by way of example- and I guess kind of going to the other point I was talking about- there was like- but so nothing there- there's only it's kind of like running um dnf or yum. You know kind of the auto updates right like on your laptop or whatever.

A

Is that it only automatically rebuilds for security uh reasons, um feature reasons are always driven by a human. um You know early st in the large majority I mean. Obviously some engineering team may have some automation that says whenever they do a release it. It goes out, um but long story like the as far as the infrastructure is concerned, for you know kind of the catalog.

A

The only time you see releases that are, or the only releases that are automatic are ones that are done by uh because of a security uh update. Essentially, the reason I was asking is just um there's. No, I don't know, I'm just gonna make up a day here right but like there's, no weekly update, that's automatically going through and like rebuilding everything, um and so.

C

They're event driven.

A

Right, okay, so so as a consumer, I can also be event driven in a sense. I don't need to go and pull how often the container is changing. I can I can feed off of the errata or whatever myself as well yeah.

C

In fact, there are a couple ways that you can you can sort of be more proactive about that without you literally having to go look at the grid, one is in the container catalog. You can sign up for vulnerability for rhsa notifications, for the images that you're interested in um oh.

A

C

A

The link in it I of course asked additional questions.

C

Yeah uh we can, we can post it later, but yeah there's a way to sign up for our rhsa notifications. But this trust me trust me from experience, because when they rolled this feature out, somebody decided to sign me up for all of them.

C

This is not scalable, so if you are a heavy container user, you don't want the feature right.

A

C

Or at least you.

A

Don't want to go into your to a human red email address right right, the.

C

Bet the better thing to know is that, if you're using openshift, um if your image streams are based on tags, then there are these floating tags inside the container catalog that you're going to want to use right. So if you, if you look down at the tag drop down again on like the row, the ubi 8 that we were looking at yep there are there are these blue bubbles around some of the tags and those are what we call floating tags. Those are tags that get moved um when a new image is produced.

C

That tag moves so latest always points at the latest uh floating tag right all right at the latest container. Sorry into that repository um 8.4 is, it is going to point at the latest release of 8.4. In that repository um you could choose to use either one of those um when you're doing like referencing a build or something like that in openshift, and you will continue to get um like updated images all the time.

C

You have to be aware that if you use the 8.4 tag, which I do not recommend at some point, it will become deprecated literally the day after 8.5 shifts it will be deprecated well.

A

This is the from there.

C

It will, it will start to decline. The health index will start to decline right so for for things like a ubi 8 image, you would, you would generally want to use latest because that's going to be compatible throughout its lifespan, other repositories that might not be the same right there. There are also these things. We call multi-stream repos and then an example of a multi-stream repo would be like if we squished the ubi seven and eight repos together and we had like.

C

We had like a ubi, seven tag, an ubi 8 tag, and you would want to pick one of those depending on which one you wanted we have ubi is not like that, but we have other repositories that are like that, and choosing the right floating tag actually gets you a different major version of software right. So it's good to always look inside that that drop down and see what the situation looks like in there with voting tax.

A

Right yeah, I'm always on the fence about um you, know what to choose here right, because so um you know, as a you know, a many year developer and to some you know, and I've played you know I joke around about. I played an administrator on tv. um I I don't like things changing underneath me right, so in that argument and and kind of in the container immutable world, I don't like things changing underneath me without kind of being aware of it, um and so in that world I want to choose the the sticky version.

A

That is not a floating tag right. I want to know exactly what I'm getting the thing is is that when I you know, I say that, but then at the same time, this is kind of where you know do you. Do you trust your vendor in a sense- um and you know, red hat in particular- is pretty good about uh not shipping backwards, breakable, changes uh and or you know, and certainly makes promises about that, it's a bug or whatever.

A

But you know whether it's I can treat that as a bug and file a ticket and get that fixed or whatever doesn't keep me from having an outage. You know um so so I'm always on the fence there. But I think this goes back to a little bit where it's kind of like you want to choose your tags and how what you follow automatically somewhat based on the provider of the content.

C

I would actually say that you, you should really think about what your use case is when you're choosing those tags.

A

Right case scenario.

C

Is you choose the tag that keeps you up to date automatically right? Because if you don't choose that, then you have to do something and we all get busy and forget, and then now you have this super out of date, image sitting there. So what we should think about is in what circumstances could the changes between versions of rel caught me problems?

C

Okay, um the binary compatibility between rel point releases is extremely good, but if I was going to worry about it, it would be in the case that I'm doing something like uh running low level c programs that interface with operating system. You know potentially, like you know, pretty low level parts of the operating system or, if I'm using hardware access. If I'm doing uh things like using ai accelerator hardware, like machine learning, accelerator hardware directly from a container, you know drivers and things like that.

C

That's when I might say hey, I might run this through a test environment, but on the other hand, a lot of the container use cases we see are using things like java and python, which are interpreted languages, they're, running business logic, which does not care anything about the underlying hardware. Code's, not even compiled it's interpreted.

C

You know- and I I can say that um in about four years of running a pretty wide fleet of python apps, um using the using the latest tag with rel containers, I've never had a problem with rail updating the.

A

C

The only problem I had with all the apps in all those years was one time when sql alchemy decided to update to a new major version of sql alchemy and it broke some stuff and we ended up pinning it to the previous version and that's it. So uh it's really it's really quite safe. It's a it's! It is a.

C

It is a double-edged sword right, like you, can literally free yourself from patching cycles in large part, uh which we all would like to do so.

A

B

A

Yeah, I was just gonna go ahead and ask the ask your question.

B

I just wanted to ask- and I think in coinciding with that, but also jumping back a little. It sounded like you said that the ubi images are only updated for errata patches and no.

C

That's not no, no, no.

B

C

The patches are the ones with the epoch stamps the ones without the epoch. Stamps are our our manual releases cut by the rail team to provide something.

B

Got it so is: is there a scenario where, when you're building your container, you would want to do like a dnf update, dash y or something like that was more or less my question.

C

C

You can do that. Is there a situation where you would want to do that.

A

I I think the short answer is no.

B

It's gonna be like a no. It's.

A

A it's a really weird use case. It does happen.

C

A

But yeah it's not like. I mean I've. Actually, I've done it a little bit with. If you have something. That's like you are relying on a very, very new thing. Then I have to do a dnf update, because I need to get something that's later than the latest rail 84.

A

Whatever image that's been shipped but like generally speaking, if you're doing that, you're probably doing something wrong or you're, it's not necessarily even wrong to brian's point earlier right backwards, compatible changes or incompatible changes are pretty rare, and so it probably won't hurt you in fact, but it's probably also not valuable.

C

I would say it's another one where I would say generally the answer the general answer to this is you should probably not do it um if you know a whole lot about? You know your specific runtime uh situation, and you know maybe there's some like moderate patch or a bug fix that you really need that isn't available, yet it it is doable right. The downside to doing this is like a blanket policy across everything.

C

C

Sometimes rpms need tweaks or some kind of modification in order to run properly in a container, particularly the kind of containers that we build for openshift, where we have to assume that they're being used by an arbitrary user id and that they're running inside of, like the security context, constraint, jail right and the general rpm available for red hat enterprise, linux or ubi, and installing into the container.

C

In I would say this is not the rule. But in certain cases it will produce a different result than the one in the container and it will not run the same as the one we ship in the container, because there's some like extra configuration, tweak being done to make that run properly inside the security context, constraints.

A

C

So if you, if you were gonna, do that you would have to test it ahead of time, make sure everything worked. The way you thought it was going to work, and I would only do it if you had a real good reason to do it, because it could certainly cause some issues. Yeah.

A

Well and kind of going back to the earlier point and related to this. One too is like, where you're doing those kinds of things. So you know, as you were kind of giving the example. You know you're doing machine learning. You know that's leveraging the driver coming out of the container. You know and you're doing something really weird to it or whatever, and so you don't want to kind of take those automatic updates.

A

That's those are the places where it's really important to invest in automated testing, and then you do take the updates. You take those automatic updates, but your testing does, the you know: does its run through or whatever um and before it actually gets deployed. So even if you don't feel like, you can put the effort into a full like ci, cd kind of infrastructure, um what you can do is say: okay, I have these they're. You know they're they're grade a you know aaa apps right. They have.

A

um You know they're mission critical, for whatever reason they are um also highly. You know uh volatile right, they're, really close to the operating system or they're really close to the tools in the operating system. Maybe focus your energy on automated testing and automatic release cycles and that kind of stuff on those components- and it's probably you know, it'll, probably be less effort and money and all that other jazz than you know doing your whole infrastructure. But at least it will cover the broad. You know the the worst cases in a sense.

B

So uh two things um so one uh just a five minute warning just.

A

B

Yeah and uh two we do have a question from murph. uh So what is the process? Or what process does engineering go through after a red hat security advisory is released for an application? uh I noticed that it does take longer for an image to be updated with the patch for the vulnerability versus what's already available via the software package.

C

uh So I assume he means it takes longer to get the container image than it takes to get the rpm, uh and that is because that's because the container image actually has to wait to be built for the rpm to become available. So um the rpm is actually the fix that we're applying to the container right. So.

B

It'll always follow and is there I, I think it's more like what's the timeline and is that an extended timeline.

C

uh It depends on the criticality rating of the cv generally with a important or critical vulnerability. We will, we will essentially have we have to wait for the rpm to come to become available. So that's up to rail engineering to create you know, create a fix and package it and test it on rel.

C

Go through all that testing for the package and then push it in the rpm repo. When it's pushed into the rpm repo there's a bit of metadata that goes with it called an rhsa, a red hat security advisory. You can actually see those on the website. um They say you know. If you search for the cve cbe dash 2021, you know blah blah, there's a number um there will be an a corresponding rhsa for that cve when a fix is shipped the shipping of the metadata.

C

The rhsa is what triggers our rebuilds um and so for for, like important or criticals. For example. Freshmakers should pick those up and rebuild them pretty quickly.

C

um If, if it's a cve that affects our entire portfolio, then and and a lot of times when there's a critical real vulnerability, like you can imagine the stack of software right if a critical vulnerability comes out for rel, then or or ubi, then then everything else we build is on rel, which means literally every container in our every repo in our container catalog needs a new image, and so what happens in that case is that freshmaker basically goes queues up a zillion builds, and then we have to start working down against that cube, um and so it can take a little while, but it usually doesn't take more than a day or two.

A

Yeah and obviously the um the ones that are lower in the stack will show up before the ones that are higher in the stack, so so a ubi 8, you know update, should land before you know an http one.

C

Yeah, so if you can imagine that you're using.

C

Let's I'm gonna, try and think of it. This might be a contrived example, because I don't exactly know how this build but follow my example. You can imagine you're using an openstack cinder container from us right then, when the rpm ships ubi 8, needs to rebuild and when ubi 8 ships, then we can rebuild python36 on ub8 ubi8 and when python36 is done, rebuilding then we can rebuild openshift cinder container because it requires that right, so actually freshmaker wow. It sounds like a little.

C

You know simple program. That just goes. I need to rebuild all these things.

C

It actually has a bunch of internal dependency logic that before it can start rebuilding it actually sorts out in what order everything actually has to be rebuilt and that's actually, the trickier part of just building this stuff is figuring out what order everything has to be rebuilt in, and so yes, if you're using ubi8, that's going to happen really fast and if you're, using something that's like way up at the top of the stack that's going to take longer because all the things below it have to get shipped first.

A

It also um is, like I said, ralph being special, um so if you, if you are unfamiliar with his work, he's done a lot of really good pieces of software, but it the other thing that you have to do is once you do the dependency graph, then you also have to um collapse the combinatorial explosion, so that you're only actually you know so you want to to not only do figure out all the things that have to be rebuilt, but then try to figure out how to make it.

A

So you only rebuild each individual thing once, um and so it is a. It is a challenging problem. um So, let's and.

C

Freshmaker has a team around it. Fresh maker has a time around it now and uh ralph.

C

Used to call devops, which would mean more stuff to our audience than uh we call it ext now, but uh ralph is the exe cloud architect and we have a team that maintains freshmaker and it's constantly upgrading its capabilities. um They've recently been working on uh making it able to do automatic rebuilds of operators as well, so that when, like an operand image gets rebuilt by freshmaker due to a cve, it can actually find the controller and bundle images and uh update their references there and reshift new version of the operator.

C

That includes the patch topper.

A

Yeah and sorry to be clear, I don't know that he uh he may have written some of the code, but I it was.

C

A

Yeah, it was his name of the the.

C

I know he started that yeah.

A

uh When he and I were working on modularity originally, we were, we uh had to name off a piece of the infrastructure as well and uh that we couldn't really talk about or whatever, but it was. uh We ended up just calling it pixie dust. uh This is where the pixie dust happens. um Freshmaker was not in that bucket um and uh but it's, but it kind of was so uh yeah. It was pretty funny. So uh let's pause there for a second.

A

uh Let's do uh let's see if I can just reuse the same window. If I can get my sizing good here, but we can pause for some sweet sweet internet points which I'm not sure if brian is familiar with, but nope. We uh if we uh give out some sweet, sweet internet points for coming to watch the show or participating in the uh the repos and stuff, and then we talk about who has won or who's the leaders in the sweet, sweet internet points.

A

uh Right now we are up to fifty seven hundred points for narendev and netherlands. Hackham uh is how we pronounce that at fifty four hundred points um and noah friction and joe fuzz still holding still at 4, 000 and 2300 respectively, uh detective kono, kudo and bacon fork steadily rising up through the ranks and if I can find my uh my g edit file, not my notepad but my notepad, um I have links to where you can get points for today's episode, um but uh yeah.

A

So that's kind of our current status on the points thanks everyone, as always for participating. We really appreciate it. We have closer promises on uh my our running joke of. uh While we understand how important the intrinsic reward of getting enough internet points is to everyone. uh We're hopeful that the extrinsic rewards are going to land soon and some of the leaders may have already received some extrinsic awards uh already kinda sorta, um but we have.

A

um uh We should be able to make more stuff available very very soon, now uh yeah, it's kind of driving me nuts. uh So there is our sweet, sweet internet points um and uh yeah, so the rewards uh are very likely. I'm not gonna, I'm not gonna, say exactly, but um basically the idea of the rewards is going to be kind of like accident. You know kind of for lack of a better term gift certificates to the brand new cool stuff store, but we've had so many challenges which I think this is like.

A

Our third brand new cool stuff store in the life of the show like so that's for the huge challenge and then t's and c's, oh and then there was a pandemic if anybody caught that on the news um so like it's just been like one thing after another uh trying to make these things happen uh so, but we think we're very, very close, uh so tick, tock, tick, tock.

A

So from here I did want to say um you know brian, what is kind of the future here. What's? What do you think is the next big step for the health index for for, like keeping you know, keeping our our containers fresh and- and you know, safe.

C

So, on the building side, like I said, uh the fresh maker team is working on automatic rebuilds of operators. That's about done, probably in a matter of weeks, it'll be in production.

C

On the data side, um the different team, the team that works on. uh We have two different teams that work on claire. We have a upstream team and we have an internal team. The internal team works with with my folks, and we are the ones that make make sure claire is able to use all the red hat data sources and we're constantly thinking of new ways to work with security, to produce new data and have claire consume.

A

C

To produce useful information for folks, so the next thing you're going to see on that front is that the container catalog on that vulnerabilities tab will start showing data about unpatched vulnerabilities. It still won't affect the grade, but you will be able to see all the um all the vulnerabilities that are in the container that don't have a fixed available as well as the ones that do have a fix available.

A

Cool okay, so so kind of like that we were talking about those two windows earlier. The window of cvd has been released, but patch hasn't been released. Yet.

C

A

Okay, gotcha, oh so, that'll be good.

C

From there we uh we're gonna start looking into grading and uh vulnerability assessment, we're already looking into it, but uh like we will, we will start producing oval feeds first with security that, um in regards to non-rpm content that shipped um as we move, you know further and further into container space. It's like there are. um We start to see packages that we just never shipped as an rpm and they only get shipped in containers. You know, and we call that container first content, meaning that we didn't produce an rpm installed in the container.

C

It's just there's content right into the container. Oh.

A

C

And we will have, we will have direct oval files uh available that describe that content. That.

A

C

Slightly different than the rpm.

A

Yeah, it's kind of uh it's similar in mechanism right to the challenge that we have with building like java things for windows like jboss, um you know so so sometimes we produce content. That is not in rpms, um and you know, containers are kind of the latest iteration of that.

C

We're working on formalizing how that works internally, you know for a very long time, 15 20 years everything has been focused on the rpm being the unit of software that gets shipped right. Rpm is the thing we ship. Rpm is the thing we version rpm's.

C

The thing we update rpm is the thing we provide source for uh you know like everything was so considered rpm, and when we look at the container ecosystem, we've actually had to start to figure out how to reorient a lot of those processes around containers right, for example, for example, when we ship containers, we want to be able to ship all the source for that container as a unit, instead of you having to go and figure out like okay. How do I find you know the source for 172 rpm.

A

Right right so.

C

We have this: we have this whole engine that we built inside container factory that produces source containers.

C

So if you go find a an image on the container catalog, you can actually find the source container for that image as well in a lot of cases increasing all the time how many of them are available and you can pull a container which has all the source available.

A

Right yeah um yeah is it click.

C

On the click on the click on the get this image tab.

A

Yeah right, you'll see get the source yeah. I was like, I knew I'd seen it yeah um yeah, which I I think is really cool right, because I mean you know like this is one of those there's all these. uh You know kind of things that are ancillary to open source right. It's like open source source is available. You can fix any problems. You have well except you can't right, because you you don't have enough context, you don't know how to do a rebuild. You know you don't know how.

A

Whatever I mean, I think one of the things that I really appreciate about red hat. A lot of the time is, you know, for the vast majority of cases. We follow the spirit of the thing rather than the law of the thing or the the you know. Literal of the thing you know, so we don't. We generally speaking, don't just release source code. We generally release kind of enough to get you where you want to get to with the source code.

A

I don't know, um but I think we do that with a lot of things and it's something I appreciate um cool. So is our you know like? Are you getting feedback on this stuff? Is this stuff you want more, you know, community participation in you know is there. Does the upstream have the same goals as we do for the container health index in a sense.

C

uh I'd say the upstream and our internal are converging. So um the back story here is eclair was originally developed by core os.

A

C

uh You know core os was bought by red hat and when we purchased core os it did not have very good ability to do vulnerability. Detection for rel there was a centos plug-in. However, it didn't work, it didn't work very reliably and it used it produced a lot of false positives.

C

um So at that point like two different streams sort of happen so, like I said, there's an openshift engineering related team that that works on upstream, clear and um kind of is concerned with the future of clear and how it integrates into clay registry, and things like that.

C

Then we have a an internal exp, devops team who works with the container factory, and we I started that team off the side, because I knew that we would need to have somebody who was more focused on making claire's uh plugins, which are not plugins anymore, but at the time they were plugged in, but making claire's ability to work directly with red hat content to be like the best right we wanted to make. My goal is to make claire into like essentially the reference implementation for security scanning for rel.

C

We don't sell claire at the moment. We don't have a subscription. You can't get a supported version of it, um but what we have now is uh there's a security scanning certification program run by our partner team and there's some links to that in the doc that I gave you and our internal version of claire is used to provide the vulnerability data in the container catalog a wrapper around. It is used to produce the grades, and the data from it is also used as the reference data for benchmarking.

C

How well other security scanners are using our oval data to identify vulnerabilities to make sure that there are not false positives to make sure there are not false negatives spoken right right. We have always had a history of having lots of false positive problems in rel with external security scanners because of our backboarding practices right right. So uh we we have to ship some certain version of python forever and ever and ever uh because what that's?

C

What we baked in the operating system we're going to support it for six eight ten years, depending on the version of rail you're using um and so when a patch comes there's an upstream patch in a newer version of python. But we have to backport that patch to the version that we've committed to support forever right and a lot of security scanners will just take a version um like a.

A

Version number.

C

Version comparison and sale, the one in real, is less than the one up stream, where the fixes, so it must be vulnerable, but what they didn't do is they didn't look inside our oval files and see that we backported the fix for the patch to some verb right?

C

That's that's layer like layer, one of where the confusion comes and then with relate modularity, which you know something about um that actually kind of becomes even more problematic, because inside the operating system might be multiple versions of the same thing that got shipped and even for people using the oval files.

C

uh It gets it tricky you have to. You have to be very like specific, about making sure you're looking at the patch stream for the right modularity component right right, because there can be like multiple moving streams right and that was actually something that even caught us off guard when rel h shipped and we took us a month or so to get it fixed because we actually had to enhance the oval data stream to include more information than it was currently carrying, and then go back and amend the clair um data consumer uh part.

C

So that it could pull that data and use it to figure out which part of the real modularity packages were vulnerable and which ones were not so you know can be complicated and our goal is one to make sure that we never have surprises like that. Modularity thing, or at least, if we do, we know about them right.

A

Right and then we know how to fix.

C

Them we know how we know what data we need to add to the oval streams, and then, by doing it, we learn how to teach other companies who are doing this. We can teach them how to do it. We can provide test case containers benchmarks for them to use just to to figure out if their product is working right um and.

B

C

All what that security scanning certification program is about.

A

Right, okay, so so in a sense right, it's it's not even it's, as you say, right, it's almost more of a reference implementation and yeah we're going to provide. You know some of that information, but you know there's a whole field out there of companies that we don't really want to compete with. We just want them to um look at our stuff and be accurate right, um and so that's that's kind of cool.

C

I mean the landscape is changing a little bit and I can't comment much on this because I don't know but like we bought, stack, rocks and stuff is a security product. So there's we have some some overlap in that space now and actually stack rocks has some components built on top of claire as well.

C

So all this work finds its way upstream and finds its way over to stack rocks too, but uh right, but right, but from for for my part in this, my my team is focused on uh making claire be absolutely bulletproof in being able to identify. You know missing patches vulnerabilities in real shift content. The upstream team does a lot of different stuff. They work on integrations with clay. They work like they. They look at how to identify vulnerabilities across different uh distributions.

C

They can identify vulnerabilities ubuntu and they look at like upstream python packages and things like that. My team is focused only on rail content and making sure that that is like airtight.

A

That got you, I got you, uh so we probably wrap it up here, we're a little bit over time uh as we often are, um but uh you know thanks so much for coming brian. uh If we have any further questions or you know they come up or whatever uh we will maybe have you back or we will try to ask them individually.

A

As you keep referencing, we have some kind of notes of links and that kind of stuff, as usual I'll dump those in the further reading section of the show notes for this show. So if you're interested in any of that content or you know or want to follow up, you know everything we kind of mentioned in the chat we'll also put there. uh So I will be putting a link up to that when I'm done with it at the next episode.

A

uh Usually I tweet out when they're done, um because it'll happen sometime between now and then uh and uh but thanks so much for being on the show. We really appreciate it.

C

Thanks a lot linda.

A

Yeah and uh andrew, thank you for being my co-host again and uh minding the chat, especially given our challenges with all of the things, and I think that's uh that's it for the show today.

C

C

C