Red Hat OpenShift GitOps Guide to the Galaxy | Red Hat Livestreaming, 8 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps Guide to the Galaxy (E45) | GitOps backed RBAC and Incident response

Description

Hilliary brings back SRE guest Jesse Sarnovsky to discuss SRE’s GitOps backed RBAC and Incident Response.

Want to learn more?
https://kubernetes.io/docs/reference/access-authn-authz/rbac/

Try Red Hat OpenShift: https://red.ht/3sMaW7O

Follow us:
Hilliary Lipsig https://twitter.com/Int2Caffeinate
Jesse Sarnovsky https://twitter.com/jessesarn
OpenShift https://twitter.com/openshift

Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1

What is GitOps Guide to the Galaxy?
Every other Thursday at 3pm ET host Hilliary Lipsig and guests dig in to discuss everything in the GitOps universe, from end-to-end CICD pipelines to creating Git workflows.
Learn GitOps https://red.ht/3saqlzS
What is GitOps? https://www.redhat.com/en/topics/devops/what-is-gitops?sc_cid=7013a000002qG1BAAU

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3LGCIeD
OpenShift on GitHub https://github.com/openshift/

#RedHat #GitOps #Kubernetes

A

A

Let's get into the big topic of open source, something that we.

B

Actually, have.

C

B

All right, uh sorry for the late start um and good afternoon get options. um I'm very tired. Today I have to jessie's gonna make fun of me. My brain is not doing the brain things um last couple mornings: I've had 6 30 a.m, meetings and I'm just like.

B

We had four hours of me being smart and like our limits exceeded, uh we just we're almost down to just like delirious and silly now, um I'm personally a big proponent of the afternoon nap like I love an afternoon nap. So I will do that after this, uh but it's a little hair early for nap time for me, yet um it's even too too early for uh nap time for jessie's cat who was awake in the background there um anyway, uh today's episode, um I've I've, talked about this a little bit.

B

We've we've promised to talk about it at length, uh so I brought jesse back to um keep me honest and we're gonna talk to people about how red hat does sre and how git ops is involved in that, um or rather how we leverage get ops for for it. So I'm gonna talk about what uh my former team uh used to do talk about what some of the sister teams used to do, uh or actually still do rather um and the cool part is a lot of our get-off stuff- is actually open source.

B

So you we're actually gonna, be able to show you like, where you can see some of the the how we do what we do. um I've talked about this before s38 red hat, operates under restricted our back. So we're going to take you through that, um but before we get there um just a reminder that argo khan is in mountain view, california september 19th through 21st.

B

I will be there um and uh other other red. Hatters will be there as well and uh just it'll be a fun, get ops related convention, so uh hi christian christian will be there right. Christian will be there, so we'll get uh we'll get the uh the dynamic duo back together, we'll take uh we'll take picture or something uh prove that we saw each other, and this is cool. This is this is local to me right. I don't have to travel for this conference uh far.

B

I have to drive an hour, but um you know that's uh that's life in california. You have to drive everywhere um so yeah uh jesse. Just for folks who may not have seen you last time or may have forgotten, uh go ahead and introduce yourself.

A

Hi I'm jesse cernovsky, I'm the team lead for cloud services, sre at red hat uh working for the same team. That hill- uh formerly, I guess, worked on.

B

Yeah, I'm still I'm still kind of between the teams right um it but yeah. I think at this point, like I posted on linkedin, we're down to the last few weeks of my transition from cssre into the um strategic enablement architecture, group or c, um and if people follow me on linkedin or looked at my linkedin you'll notice that I've actually posted my job title is chief mermaid.

B

uh That is my actual business card title until they make me change it, which I don't think they will. um But they said I could pick my business card title and I felt like chief mermaid has always been an aspiration for me. um So I naturally chose uh chose that as my my business card title and uh so chief mermaid, I am in the c-suite yeah yeah. I did that you're welcome and um yeah, like I say just you know, navigating the choppy waters at sea.

B

um I also lean very, very heavily into every single, like nautical or pirate pun that I can related to um related to this. Wait, I'm sorry who will be my vp of choral relations, no comments um so uh yeah. So um that's! That's! That's! What's going on in our side of the house- and uh I already truthfully reorgs- are always a very um a very interesting time or be being a real rally.

B

Rather moving organizations is always an interesting time, um so the I feel like I'm already so out of touch with what I used to do and I'm still technically doing it day to day so, um like I said jesse's here to keep me honest. That way, I don't forget any of the important things about how how esserie has been using git ops is very technically I'm not going to be an sre anymore, but I still don't have a better job title than principal reliability engineer um because I'm still doing reliability.

B

Engineering, it's uh just on the like the before before the pager exists side of the house. So uh that said, um let's just get to it right.

B

A

B

um I'm actually gonna share my screen, something I don't like ever do I've noticed on this uh stream um and we're. Actually, I don't want to start on that window. We're gonna start on this window. Okay! So, uh let's, let's talk about our back real, quick um right, roll based access control um and uh jesse I'll. Let you keep me honest, like I said also, I maybe be the smart person in the room today, um I'm just a mermaid and a fishbowl, that's what's where we're at.

B

He hates me so much right now. He has been putting up with this for weeks. By the way um I am shameless. I have perfected some of my puns um over over the last couple weeks. Some of them started out like less cringy, and I figured out ways to make them more cringy. Oh yeah swag. I forgot to talk about our swag. Just real, quick um red hat did release uh get up to go to the galaxy swag, and uh so here's the coffee, mug oops uh best part about it.

B

Is it's like the picture on the both sides?

B

Why is that the best part, because I'm left-handed, and so I'm always holding coffee mugs, and you can't see the design on my coffee mug it's facing me, uh but when, when folks print the designs on both sides of the mug, then when I'm holding a coffee mug, the design also is outward facing, um and then we also have our t-shirts.

B

uh These exist in the cool stuff store for folks who are interested in purchasing their own. um I've heard rumors that they are being given out at conferences, so the coffee mugs were at vmware explorer um in san francisco. Last week, so if you're attending a conference- and there is a um red hat presence stopped by the booth- maybe- and you could luck into some swag um but yeah. So since I now remember, jesse picked up his coffee mug and had a drink, and I remembered I forgot to talk about the swag.

B

I also am very tired. I think I mentioned.

A

The sixth time I think, yeah I'll keep.

B

B

There's not enough coffee in the world today I have like the shakes and it hasn't helped me wake up at all at this point, so we're just this is where we are, um but I did want to talk about about. uh This are back thing right, so role based access control, and um so what this means is uh we're sres for managed services running on openshift, dedicated right, openshift dedicated is a managed version of openshift um openshift being red, hat's opinionated kubernetes, one of the biggest opinions that openshift has has to do with security right.

B

uh You can't run like pods as root, for example, like the security and openshift is very, very, very big built-in opinion, the security contexts. So um we we kind of got continue with that uh opinions into even our own production access when running production support, so we actually define our production support access or are back in in git right and like I said this is a public repo.

B

If you want to go, look, you can just go look, um and so then this defines and who is in these groups, is like that's on the like locked down inside of the house. You can't see that um who is in these groups the actual management of these groups that are defined here. All of that is is done internally, as well, also through git ops that links into some other systems. That way, if somebody leaves their access is revoked, um but the the general like who has access and what do they do.

B

You can actually see it here and it's all declarative right. We define it, we define it with some yaml, um I'm trying to think about what is the best.

B

One of these to look at um right, and so it basically just have levels of our permissions what we actually have access to, so we're actually namespace restricted in our access, as well as action restricted.

B

um So that means not only can I only really see certain namespaces, but then within those namespaces. I can only do certain things.

B

That's another one, hang on sorry, I'm looking for the one that had all.

A

B

Yeah, where, where are those verbs.

B

I never remember it off the top of my head nope that one's also good though, uh but anyway, you know I've had to modify these so much you would. You would think that I would actually remember where the verbs are, but no no, I don't.

A

So essentially, in these, in the files or or like she said, name spaces or a name space pattern to match uh verbs is in actions like we do gets. We can do uh you know edit uh and then there's targets which uh relevant to those. So you can only access get pods get staple sets. Something like that sure.

B

There's a yeah yeah. I know right, I'm like, where are the verbs I felt like? I would be able to very quickly go to it and now I cannot remember right um if we have to do anything, um so this is actually a name space for us if we have to do something like spin up a special debug pod or anything like that, which is a super rare occurrence, but has happened in the past. This is where we're allowed to do that.

B

um So it again again, all of our access is very kind of it's very strictly defined, um but occasionally you have to do things that require more access than you have by default right um and so we've solved. For that too, um this is called manage. Scripts and uh backplane is our method of accessing clusters. um That's just the name of the tool, um so this actual set of scripts.

B

It enables us to have access um that might sometimes exceed our back because it doesn't actually. So the script has a higher level of access than the person using the script right. So we can write scripts that enable cee to get things like additional logging. They may not. Otherwise, have by default and there's like a process that the logs can go through to to like obfuscate, anything that would be particularly sensitive, but then it doesn't actually increase the cee production access.

B

They just have access to the ability to run the script and again people who are in the cee group the people in that group that's all audited and everything and get optified off in the often the the secret side of the house right. That's that's behind the protections of of red hat, um but, generally speaking, you know we like to think that we have put in some best practices um into our into our approach and we'd open source like everything right it's. I was honestly a little surprised. I was like plotting this episode.

B

I was like how much of this? How much of this can I very easily talk about without having to go into just straight?

B

Only talking about it conceptually and the answer is, I can talk about most of it explicitly because it's all just here available on the on the public internet um and I've actually sent this um off to folks uh to show. um This is just an example of a script um that we that we've put out, um and I've actually actually sent this to to folks, to show to um to show the customers too, because sometimes our customers have that that question like okay, I see this.

B

This comes with sre support, but what can sre do right and it is somewhat defined in our our our documentation that we show, but it's a little less explicit, and it's honestly like you, could you can see what esri can do? You can see what what we might be doing and how we might be doing more or less how we might be doing it, um because we we let it all be. You know, open and available to you to look at so um that I think goes along. It's part of our.

B

I think the general red hat strategy of transparency and building trust um so we're not afraid to basically say hey. This is how we've done things and we're not afraid to change things. If um we think we can be doing things better right um and jesse you've been at red hat seven years.

A

Seven and a half almost, I think, yeah.

B

A

We're just gonna.

B

Round to seven um so you've been a red hat seven years, so you've seen this more than I have right, but we didn't used to some some years ago. Our method of accessing production was actually different right.

A

Yeah, particularly back with openshift dedicated three, um the rules were I mean we didn't, have the the professional backplane concept and stuff like that, so it was. It was much different before, um but when it came to openshift for we, you know this. This stuff is honestly monumentally better completely logged everything's there auditable um yeah highly highly sophisticated compared to what it used to be. I guess six years ago.

B

Yeah um I yes, uh I had the very interesting experience of because of when I joined red hat and the team. I had the very interesting experience of learning, openshift, 3 and 4 at the same time um now that now that we're not actively supporting anything in openshift 3 anymore, I I fortunately have been able to better create separation in my head, but I you'll remember from working incidents with me a few times jesse, where I'd be like wait, a minute can't we just do like xyz and you'd, be like no wrong.

B

Openshift I'd be like dang it. You know.

A

Yeah yeah products yeah, I had different versions. For I mean we had an overlap of what two and a half years, or so we were supporting, managed applications on both those. So.

B

Yeah yeah, the entire first year I was here, was still fully into supporting both openshift 3 and openshift 4 um every day. So, and and yes, uh I will say the thing that sort of made it easier to keep track of which had which features was the fact that we had the better access model for openshift4 right. That we've had this um kind of some important cool things about this.

B

um We use, we reuse service accounts right, um so all of our back is tied to service accounts, and um that's that also allows all of our uh auditing to be um to be tied to the same service accounts right.

B

So uh we're able to define these service accounts and then authenticate as a as a person who belongs to that service account um when I, when I, when I'm as myself, and that checks that I'm somebody who who deserves to have like not just who am, I employed at right ahead at who is active, but also an employee at red hat who's active who is supposed to have this level of access, and that I like am, who I say I am because I've gone through various label layers of authentication right.

B

I have to authenticate a couple times and and dual authentication, um so that's all like very cool, um but we're not we're not using anything extremely on uh like unusual we're, not adding anything. It's super special to the mix right. This uh service accounts are just built in so we're able to just like leverage openshift itself like in ways that almost anybody could do you could you could realistically recreate our pattern. Here um is what I'm saying we haven't.

B

We haven't added anything um special to openshift dedicated to enable this that you couldn't do yourself, um so yeah manage scripts. What do these do right? um I want to talk about the advantages of manuscripts uh and jesse. I think you've been more. I feel, like I kind of handed down the we're doing manage scripts to the team and you actually had to like deal with the implementation of what that meant to do the manuscripts to the team right. This is this.

B

Was this was a set of way of building and standardizing tooling, across red hat, created by the team that did back the backplane access um so like big shout out to them by the way, for the really really really big efforts here like this was this was significant work on their part. They had to talk to a lot of people and learn from a lot of people, and I showed up to their planning meetings and ruined their days. Sometimes by like saying bad news: that's not going to work for the layered products career.

B

If you need to change things like um huge shout out to them for the sheer just volume of people and use cases, they've had to figure out how to accommodate in doing this.

B

For us and the fact that they've been able to keep so much of it still open source while doing that so like um I don't want to embarrass people by calling them out by name, but I'm going to do it anyway, uh so like kristoff and haran are like um just like off the top of my head, like two two of the most important people for the back queen ever but like way more people than just that and they're gonna go like find out the hard way somebody's gonna tell them.

B

I mentioned them now, but I'm not going to do it. So maybe they won't feel embarrassed, but yeah uh backplane team did great. uh But let's, let's actually talk about the implementation of the manuscripts a little bit because, um like we have it here right and you can actually see that we've actually created like srep, is the platform sre right when they, when they make these scripts to to handle incidents.

B

I like to say: hey: it actually can decrease our mttr or mean time to resolve, um but I'd like, like jesse I'd like you to expand on that a little bit like in terms of what it means in like the reality of incident management.

A

uh So, um frankly, yeah a lot of the things that we end up having as a manuscript or something we've actually done in production. You know, honestly, if it's. If it's happened to two or three times in production, we've had to do something or we had to escalate privileges or something or had to do some special handling.

A

uh We immediately flagged that as something that makes sense to become a manuscript to basically simplify the process. uh Frankly, some stops get to be. You know, 20 or 30 steps in some cases and putting them into a manuscript. Where you just run one thing it does: it collects all the output puts it all together, you don't have to escalate privileges. Anything like that, it just runs. Other people can run it and it's very selective.

A

So it's very important that you know these things are safe from injection attacks where parameters are just dynamically created and can do bad things with the wrong permissions. um But you know anytime, it's it's something that we repetitively do. It makes sense to become a manuscript, and I know several people on my team. Have um you know recognize that hey?

A

We should just make this a manuscripted before we even bother writing a stop, it's easier to make a manuscript for this, but we do have the stops to back it up in case there's a problem running those managed scripts. So once.

B

C

Have it thoroughly.

B

A

It makes sense to turn into a script just like this.

B

Yeah, um I'm just I just pulled up this one I was actually looking for, um and I know it exists in one of these because I've used it right. I was looking for. The must gather right must gather, is probably like the the managed script right.

A

It's in cee right, I.

B

Thought so, but I didn't see it, but I could also be blind.

A

Well, you know yeah.

B

I know right, I know it's here somewhere.

A

Most common one used.

B

Most common one used, I can't remember where it is ah it's interesting.

B

Why can't I find it? Oh! Well, I'm tired, I'm not um I'm not hurting. My brain any further than this um right, but so must gather, is an interesting one right. We we call, it must gather it's basically everything we think you need to get from a cluster experiencing some sort of like critical incident right, some something something major, especially if it's not immediately obvious. What's wrong, uh like multiple alerts are firing right. Your very first step should be a must gather at that point um and basically what we do with that.

B

The reason I say that it can reduce the mean time to resolve is even though the script takes a good while to run and gather everything it needs like we're. Talking logs and it'll, get descriptions of things and it'll get like scd, health and other stuff. um The uh the most important thing about that is, is you can kind of it will run in the background right, so you can have it running in the background, while you're still actively doing your manual investigation into things.

B

So the fact, like the usage of the managed scripts, the ability to to fire them off they they run as a job right. It's a kubernetes job, so it'll run as a kubernetes job on the cluster, and then you can basically kind of check back when it's done. It'll give you a basic, a pod identifier from which you can get the the output of the script like what the logs it create, or what files did it create and so forth and from there you can.

B

If you need to copy them off into other locations, you can so.

A

Is more like a snapshot of all the things that say because, frankly, if you're looking into a new issue, particularly something you don't already have a stop for you don't know what you're necessarily gathering and musk gathers taking all that data putting it together. You might find later that you know once you fix the issue, you can still refer to those logs that were collected and actually understand what happened at that point in time. So very important process.

B

Yeah, um like I guess it does, take a while to run, but the fact is, it runs as a job and um it allows you to conti. It basically allows important investigation and gathering to happen. While you continue the manual investigation, because if you had to stop and get all those pieces by hand, you would spend two hours just getting those pieces by hand and your investigation into the incident wouldn't have have gone any further.

B

um So it's yeah, like jesse's, a very, very important first step that we do something similar. I would say like if you're trying to figure out how to do sre for the first time, like that's a great starting place as well.

B

um Did my my phone is like blowing up? Are you all hearing my chocobo do I need to like? I don't know.

A

Okay, I like to sound chokeable, though I wouldn't tell you.

B

A

Really smiles on my face.

B

uh There's definitely been past streams where somebody's like. What is that? It's like it's my my text message alert is the chocobo chirp from final fantasy, so um there are some past streams where I'm like. Oh, you can hear my phone go off in that stream uh and it's like I just like it's gone off three times today, so I'm like, I can silence it but like I also don't want to be really rude, midstream and lean down and silence my phone, because I forgot to do it ahead of time.

B

So if you guys can't hear it I'll just continue to ignore it.

B

B

uh I'm just reading our hopes, nine. Our hope nines comment, uh the five minutes when the cluster works. Fine is like more important than the five hours on a cluster where, with sdn at cd issues, and that could be a challenge. Yeah.

B

A

Just hearing fcd thinking yeah when ncd's running slow, nothing's quick to gather, I think, is what we're trying to point out there.

B

Yeah, no it's! If nct, if something's wrong with ncd right your whole api server might be unresponsive or lagging in response, and everything else like that. um Correct me very wrong. I think our must gather strip actually can account for that, though, right.

A

I couldn't tell you.

B

Some of some of our stuff does some of our stuff. I don't know off the top of my head, which ones do and which ones don't, but some of our stuff does actually account for um when there is a um some some sort of pressure, um because there can be, there still will be some retries and stuff built in. But that's the point right of of doing it in this way and having a standardized set of scripts available to people to use is that you can.

B

You can write them to account for things like uh disk pressure or api server, flakiness or some of the other things that are going on um so and again, like the the most important part about the way we do. Things is most amount of actions least amount of access. People have heard me talk about this and say this. If you've heard me talk at all, I say it all the time. um I think I stole that from somebody. I guarantee you. I stole that from somebody that that's that's not. I did not invent that.

B

I think the I'm not even sure I invented the use of um as a verb, although I think I'm credited with it like so when you have an out of memory event on it on a pod right.

B

It's like it's the um I I say it's uming and I use it as a verb, because it's delightful but anyway, um most amount of actions least amount of access, uh and the way we've done this here with these managed scripts um enables that again, everything is tied to service accounts, and these are these are mechanisms that just exist that are are totally reusable, um but there's some things that we do right with git ops that are not actually like, captured here in the public that we're gonna have to talk about like a little bit more conceptually.

B

Part of it is, though, so we're gonna go over here. um I've joked before that. You should never anger, app sre, because they'll turn off quay right and you're like who's app srv app sre is one of our sre teams that that supports some of the managed services like quay, um and they do others and the way they've done. This is through something called app interface.

B

App interface is not fully open source, but we do have some pieces of it out here for the public to look at so again we're trying to make our patterns available for people, so they can kind of see how we're doing things and I'm not saying that everything we've done is always perfect, but they've definitely been working um so quay or key. Yes, okay, everybody, I know, calls it quay if anybody else calls it key. I've never heard it, except for occasionally and like questions like this on the stream, or I think there was a.

B

I think there was a memolist threat about this, whether it was quirky. I think there was um everybody I know calls it qui uh jesse. Have you ever heard. It called key.

A

uh I probably four out of 150 times whatever so I'd say it's probably in the one one to two percentile of key and I'll be honest. All of those people are in europe that I can think of. I don't know if that's a thing or something different about europe and probably closer to uk, possibly.

B

It's a french word. I care according to to our hope. Nine well.

A

That may be that, maybe that's it, so I feel like they're in that geo. I I'm used to working with a lot of different time zones, so kind of gray area.

B

uh The first one about memolis is we don't mention memolist. I have violated that rule so many times. That's the best kept secret of red hat, in that it's, the worst kept secret of red hat.

B

um I think I actually have memo list famous on my resume.

A

I feel like I should be inquiring about that. I don't think that would kind of deviate from the goal of this uh talk here.

B

um No, we were actually putting together um uh profiling. This is like uh we were putting together profiles of people who, who have maybe like less sre, like backgrounds who have made good sres, and I've mentioned all the time that literally right before I became an sre.

B

I was a quality engineer, so I actually updated my resume for the profiling thing um that we're doing, and I actually asked a few people across red hat for their resumes as well to do the same kind of a thing, because we're trying to this goes into like the whole job family thing and um and and trying to um like identify uh how to how to continue to find great talents right. I think I've actually even had conversations with town acquisition or like hey.

B

How do we can we use the streams to help attract talent, and I think you guys should definitely join the comments of our streams, because occasionally there's people interested in how to become an sre or a developer or a system or whomever at red hat and like yeah totally. um But yes, so, every now and again I get involved in things like that, because I I really care about um about hiring and and our talent pool and so yeah.

B

I updated my resume for that and I threw that in there as a joke to see if anybody noticed um and since you know the people who went over that probably watched the stream, I don't think anybody's gonna notice, um I don't think they're reading their resume- that closely.

B

To be honest with you, I think they're just pulling out the keywords uh so anyway, uh but that's officially on my resume uh for giggles, if I ever have to actually use that resume I'll, probably take it off um so right, app interface, we're going back to the actual topic at hand. This is important. This is app interface um developers who have created, managed services at red hat are familiar with app interface, and um so it has basic.

B

Basically, the idea is that you use openshift templates to define an application, and um then it takes over from there right. It has ci cd and you can even you can actually even use it to build the artifacts, although you don't have to um because if you build your artifacts and you define your images and you put them in something like quay right, then you just tell the yaml hey. This is this: is the registry where my image is and you you give it the manifest it actually promotes, manifests right.

B

So this is this. Is this is really impactful for us, because this is the way our managed services that we sre are delivered. One of the ways that our managed services that we are delivered um and one of the other really important things about it- is that we well bishop tells what is this 2015. christian today I swear um you know I just don't I'm not even with you man, I'm not even with you.

A

B

Templates are fine.

B

Just not even uh right so the other things we do and jesse I'm gonna need you to talk about this part. This part, especially right, because I don't remember this super well already. I feel super super divorced from it. Some of our we actually can do um like cluster scaling and sharding um through through app interface and through the get ops um through the git ops model, and I just want you to high-level, walk us through what some of that looks like, um so that I don't have to be the smart person in the room.

A

Okay um yeah, so I in the app interface honestly, if you think about we, have pretty much everything modeled in get ups form. I mean users. Teams, like you said uh you know the repos being used uh the services themselves uh and, and often those services will have uh some sort of configuration file like sas kind of configuration file that defines what makes sense for that product. So, frankly, every product can be different with the values they use.

A

I mean they could be arbitrary um in some of our managed services, wherever we actually have like the individual clusters involved. For um I don't know if anyone everyone understands like control plane versus data plane, but we have a list of what clusters are involved, uh the ids for them potentially what size they are either in terms of instances of the like the instances of a product specific thing such as like managed kafka instances on there, um as well as what regions they're in even defining new regions.

A

All that stuff uh can be defined within one of the sas files and.

B

Again, they're very.

A

Product specific so um for managed kafka, it's probably the best example we'd have of where we've really utilized that that concept and and virtually everything can be configured with just a few settings in there once the cluster is connected. Most changes are done purely from there from a modification to the sas file and letting it roll out in about 10-15 minutes everything pretty much reconciles.

A

Thank you. One of those tabs you have is about the is it contract uh reconcile or contract yeah.

B

A

Thing so now I cannot speak with uh like authoritative stuff, but the contract thing or whatever is basically like uh a commitment to what the service will do or what the automation will do, based on changes made to those files and the reconcile is essentially a reconcile loop, much like an operator, um but it's working through uh various. You know: ci cd systems, jenkins stuff, like that, and that will you know like roll those changes out through various systems.

A

um Hopefully that gives a general overview without getting too complicated.

B

Yeah, it also manages um so there's certain integrations. I mentioned this earlier right. There's a standard observability stack um for apps to find an app interface right, and so it handles that integration as well, um and that's all done through the same um basic um basic uh pieces here, the the record sound so that you can actually see some of it some of some of what it does over here um it. Actually it does like a lot of things.

B

It actually does a lot a lot of things, um one of the one of the really important things to how we do everything is hive right.

B

So um hive is a tool that red hat has developed to basically um manage and keep synchronized a fleet. um So when we say that we treat our clusters like pets not or like cattle, not pets, hive enables that right. I know I mean my dyslexic self. Reversing everything.

B

Dyslexia, plus sleep deprivation means that you get all of the right words in all of the wrong order and it completely changes the meaning cattle, not pets. That's what we're on about um and hive allows us to do that right, but hive itself is its app interface.

B

Things define an app interface right, get hive actually will will do the the the synchronization of applications and so forth across the openshift dedicated fleet for things in app interface or one of our other methods of defining managed services or managed apps, but hive itself is also defined in app interface and then supported by the app sre team, as as you know, keeping it highly available as as a service that is really the backbone of a lot of this right.

B

If I, if I need to you, know even like the changes to the our back stuff gets propagated out actually through the hive um uh updating the configurations on on the entire fleet of clusters, I mean it can. Do I mean the scale of it is tremendous. I was actually just asking the the hive team stupid questions yesterday because I was like I have stupid questions because I think I know some stuff about hive and there's some things.

B

I don't think I know well about hive um and they're super nice, a very, very, very cool group of people, um and they they answered my stupid questions but um yeah. So it's it's. We we we use that we use this ourselves right um very extensively and it's that's that's the way that the um the that's the way that the the the something gets made. I don't remember the idiom.

B

My brain's dead, my brain.

A

B

There's like a somebody.

A

A

Hot dogs are made, I think it's one of the things like.

B

That I don't know I do you know what I give up stephanie's offering to save me from myself here, because I've I've lost my my expression, it's okay, it's okay, stephanie, we're fine, um but um yeah. I actually am going to pull up, though speaking of things, I'm going to actually pull up hive because it's in the open shift.

B

I got a new keyboard. It's super sausage. It's happening.

A

Thank you. Looking for a post of hot dogs.

B

Thank you, martin you're, my hero. um I can't even remember where that's from somebody else. He probably also knows that uh again some people who are smarter than me bailing me out. I appreciate you red hatters, so much um right. This is hive right, also um also public also open source, um and uh it's it's very cool.

B

It does um a lot of very neat things in a lot of very neat places, and it's pretty it's in a lot of ways: it's our backbone. uh It's it's very, very critical piece of our our infrastructure. um So hopefully the hive team appreciates.

B

They shout out here because they they do deserve it um but yeah. So this is this is how a lot of the get ops changes that we're talking about that we will do get actually like propagated to the the target clusters. It's through is through the usage of hive um definition of a stupid question. Something you are. Are you already very familiar with and obviously know the answer to that? Never saves me. I still ask stupid questions all the time.

B

I should know this, but I don't remember so I'm going to ask ask it anyway, and then somebody would tell me like oh that's right and suddenly I will have all that information at my fingertips, but my brain doesn't always do the brain things and I have decided in my old age that I'm not going to pretend to remember things or know things when I don't and I'm just going to ask and my life is much better and far more stress-free, because of that I'm telling you um right so that, basically, I think, covers the majority of the ways that we interact with git ops day to day.

B

Did I miss anything jesse.

A

um I mean we probably work in some of some of the things that you wouldn't expect. Necessarily uh that we would do through get up stuff would be like even resizing volumes for some of our products. Whatever changing size of volumes, we actually have even those defined in a repository where changes are made there and they're rolled out.

A

So even if we had to make changes like that, we would we still use the same, uh get up processes to do uh very simple changes, as well as some very complicated changes and they're all orchestrated in the that case, mostly through open shift through the control plane through openshift. I don't know other than that. I think it covered as best yeah it's hard to fit in this small slot. So.

B

Yeah yeah, and it's also one of those things where like without I don't have a great way of getting hands on keyboard and showing this um so there's you know, that's just that's just uh that's just it. um I do want to find all right. The manage cluster can pick up like going back to this. I'm like I there's actually a lot of other stuff in managed cluster config as well, not just um rr back.

B

Which is, I think, I kind of kind of I didn't really touch those at all. It's kind of that's kind of important. I got I get most excited about our back being here, um but there it is, there's actually a lot of other things that go into the manage cluster config and and into the way that we manage our our fleet um that all comes back down to these.

B

These get reposts and the the pipelines and jobs that do things based on the changes that exist here um so, like I said these, these things are, if you just go, look at the open shift um right. I think we do is open source right. Just go! Look at open shift!

B

um There's there's a lot! There's a lot here to go! Take a look at if you're curious about how we do things. um I think it's kind of like my favorite thing about this job is I'm like. I don't have to like it's very easy for me to to talk about it because of the fact that it's it's all right here um so like feel free to to like fork things.

B

If you want to right, that's what they're there for uh play around with it, there's all kinds of really cool things in here, um but, like my my favorite thing in in this managed, cluster config is actually the um is actually the rbac, um the fact that we don't just run around with cluster admin and that we have um restricted access to protect the the workloads being run on on the clusters from basically unintended like really whatever there's always.

B

You always say that you're you're protecting from bad actors, but that's that's realistically, like only half the story, the other half is that you're just protecting from accidents right, there's. Somebody accidentally doing something like uh you know, using a star regex in the wrong place and suddenly you're trying to delete things across everywhere. Instead of in like this one particular place right, so that the fact that we're operating with a restricted are back creates that layer of protection against human error.

B

The fact that we are codifying things into scripts when it needs higher levels of permissions that allows us to take actually more access away from the service accounts and adds in even better layers of protections.

B

um These are these are basically, I would say, like solid patterns, um that I would recommend you know people look into, and I have definitely been asked this question by folks before, like do do you guys use cluster admin and the answer is no: we don't use cluster admin um and on openshift there's a concept of dedicated admin.

B

uh We're not even running with dedicated admin either right uh we have our our back is, is, is custom and restricted and has some admin functionalities, but not all of them, and that just kind of gives everybody a higher degree of um confidence that no matter what we're doing we're only doing it in the places we're supposed to do it, which is great, because incidents are stressful enough without having to double check some of that stuff.

B

um I think we might end up possibly ending this a little bit early today, which uh is fine, um but uh before we do that before we do end, I I want to invite stephanie to to to reveal herself um from the background.

B

I think she said she might be, might be persuaded to to to do that for us today.

B

If I can stop sharing my screen, where is my there, it is.

C

Okay, definitely I love you I had to. I sat down for a second.

B

uh I always keep stepping on her toes.

C

Sorry, it's glitchy today, it's the okay, yeah and now I'm kind of tall. It's the last show, or my last show this week.

C

There we go kind of standing up now.

B

All right are you in the the rally tower.

C

Yes, here we go, let me change screens.

C

B

um So folks, who've been watching, the other streams have probably already seen stephanie like pop in um to those uh I'm not. I actually haven't had a chance to watch the other streams. I feel really bad. I haven't gotten there yet uh this week, um but, like you mentioned, it's actually your last. um This is our last stream with you, as our producer on get github's guide.

C

Yes, unfortunately it is, but I really enjoyed I'm so glad uh I got to be a part of hillary coming onto the show. It's been great.

B

Yeah, I would not have been able to do this at all without you, I'm still, I'm still a little scared. uh I know you're not going far right, you're, not leaving red hat.

C

B

So drastic you're just doing other slightly more high profile things these days.

C

More streaming more shows on the youtube channel, but uh you know how uh the org change every everyone's been. Roles have shifted in the last few weeks. So.

B

Yeah, okay, big changes all across the entire of entire company really, um but we're still gonna keep streaming. um Stephanie may occasionally still just like drop in the comments and say hi she'll pull a christian she'll come in the comments. Just to give me grief.

C

All right and I still have access to restraint, so I might just jump in.

B

Yeah yeah, oh man, I would I would I'd be over the moon excited if you did that, if you just like bombed the stream one day, it would be amazing um but yeah. So I just kind of like thank you stephanie. I appreciate you. You have been an absolutely amazing producer um and uh I just want everybody to just like give stephanie a little bit of love and appreciation right now, because uh she's been awesome.

C

Thank you all. I appreciate it. I love doing this, so it's nothing. It's fun.

B

Okay, I'm glad um okay, so coming up um next stream in two weeks uh will be right after argo conrad, so we'll go ahead and uh we will talk a little bit about argo con, um but we also have a really cool, um really cool next stream. Coming up. I don't know if I've talked to it all about this jesse, because I feel, like I've talked to everybody about this um and sort of emerging technology at red, hat, called uor or universal object, representation um and um yeah.

B

So so uh martin knows what I'm talking about, because he was at um dev conf in boston, where I learned about this and uh so we'll be joined by uh jen power uh and alex flom in two weeks, god. I hope I said his last name right. I should have double checked that um in two weeks to talk about this, which I think is very cool um because it's taking a lot of the principles of git ops and it's applying or like the themes and it's applying them in new ways.

B

uh So I hope that everybody will join for that stream. Learn about something cool. I think they have a demo for us. um If you go hunt down the dev conf um uh recordings from from this year somewhere in there is alex and andrew block talking about it as well. So you could get a cool sneak preview, um but I'm hoping we're gonna get to go in a little bit more depth on some of the pieces uh next time, and um so this is also I'm not.

B

I don't have any immediate plans to to drag jessie along for the ride, so I'm not alone in my fishbowl um in the near future. So uh we're also just gonna quickly say thank you jesse for making me be not alone. In my fishbowl the last two weeks.

B

I still have to decide if I'm getting a co-host or what, because I hate hate talking to myself or talking to like the appearance of nothing.

A

Oh come on. Thank you for having me these past two uh streams. That's been awesome. I find it hard to believe you don't like talking or hearing yourself talk. That's all right.

B

A

No, I don't mean that in a bad way, never I'd put another way. I've I've never seen you afraid to speak in front of people. Maybe there's a better way of saying it. Not necessarily you want to hear yourself talk, but you definitely more public. Speaking than me,.

B

um Right anyway, um so that said, um I will, if you're, going to be there in person. I will see folks at argo con in later this month. I'll see you folks back here in two weeks for our next stream jesse. I will just see you like later today or something probably uh and stephanie you're my favorite.

C

Yeah, I was gonna say I know I can come back on and you can just teach me get ops, that'll.

B

Be a long, tedious.

C

B

C

Already know how to.

B

Use the command line, or are we just starting straight from there.

C

I know how to use the command line to a certain extent. Okay,.

A

C

A

Oh, don't get us thrown on ms-dos again we got yell that last 31 flavors of microsoft, yeah.

B

Microsoft that did end up slightly happening. That's what happens when you ask people to reminisce about their first operating systems. uh That was my mistake in that in that and that prompt um but yeah um I'm just go ahead and end us six minutes early. I have no shame about that. I'm so tired! Today, folks, um I will say thank you, everybody for joining. Thank you, jesse for joining stephanie, again you're my favorite, um and I will remind everybody, as per usual: hey folks, choose your technical debt, wisely yeah yeah, bye.

A

A