Red Hat OpenShift GitOps Guide to the Galaxy | Red Hat Livestreaming, 3 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps Happy Hour (Ep 5): Operators in a GitOps world

Description

Join Christian Hernandez, GitOps Extraordinaire, for a journey through how to achieve GitOps in any number of ways. The occasional Red Hatters and special guest will join us too.

A

Good morning good afternoon good evening and welcome to another episode of get ops happy hour, I am queer short principal technical marketing manager here at red hat, I'm joined by my fellow principal technical marketing manager, or are you senior principal? I forget what you are now dude my bad joined by my fellow uh co-worker uh christian hernandez. The get ops extraordinaire uh welcome to open shift tv today we're going to be talking about operators in a git, ops world. So christian, let's, let's tell everybody what that means.

B

Yeah yeah, let's see everyone yeah yeah, so by the way. Thank you uh everyone. For last time, we had a um a good session, get ups happy hours, a special kubecon edition. um uh We got a lot of good feedback right, that's nice! To get all the the talking heads, the uh the um uh the the the thinking minds behind what what get out is gonna, be.

C

B

So uh that was really cool. So if you haven't caught that uh go ahead and go to um the youtube channel right, you can see all previous streams there and um yeah. So today we're kind of talking about operators.

B

um This kind of uh topic came up in my head here um because anytime I talk to anyone about just cloud native architecture and just in general, there's just so many um so many automation tools right. There are tools they're, like especially when you talk about operators and openshift, because it's like well like what do I need operators, I have helm and it's like well what why do? I need um operators if I'm using get ops to trigger the automation? It's just there's.

B

There's this whole, um I don't say: there's like like venn diagrams right, like it's like a it's a multi.

A

It's a venn diagram that looks like a star yeah.

B

Yeah yeah exactly.

A

Yeah, it's a venn.

D

Diagram that it's like.

B

You can use like a little bit of everything. How does that all fit in right? So, um oh and by the way uh uh chris, I know chris you're you're you're drinking an ipa, I'm I'm having wine, because.

A

B

Being from california,.

A

Yeah ah good point: yeah, I'm actually uh drinking cigar city brewing from my favorite brewery in florida. It's finally up here in michigan, so I was stoked once.

B

Yeah, it's always nice when you can just get it like locally. Instead of.

A

B

Drive to florida, yeah.

A

I ain't driving to florida, no more.

B

A

Long drive now.

B

Yeah, that's a long drive yeah. There were some breweries in milwaukee that I really really liked, um and I was like you know back when we can travel um that I'm like man. I wish I wish they had some of this stuff here in la but anyway, um back to the topic at hand. Right, I just kind of want to say we're the happy hour here um is um we talk a lot about get ops and how uh you know get ups in terms of like um no not like miller.

A

A

B

I'm a faux hipster, so you know only microbreweries, um the um you know get ups uh by pull request and um you know all that how argo and you know a lot of your sync tools. Acm flux, does a lot of the automation for you right so, but then we have operators and if you've been around the red hat ecosystem have been been around openshift a while all we do is talk about operators right, so I think yeah.

B

um I think uh it would be a good time to talk about how operators fits in in the overall get ops architecture right. So I do have presentations. So let me share my screen here: I'm not gonna, um uh I'm not gonna, kill you guys, oops. Let me not.

A

Slide us to death.

B

uh Yeah, I don't want to slide you to death, I think in the um uh is it desktop three or desktop two. What's.

C

B

I have like one two three, but.

A

B

On linux, is it's one three two like it's really weird, but anyway,.

A

Oh so it's funny story like you might notice. If you haven't watched the show recently, uh there's no flags behind me. So when I moved my desk up here, yeah like I wanted to rearrange some things right. I wanted to put the tall monitor on the right hand, side I wanted the laptop on the left, hand, side and yeah it completely. Freaked out this monitor was turned like a 90 degree view.

A

This monitor was normal, so it was like I had to like scroll and everything was out of order so like just getting it back to normal, took like like nine or ten tries, so it was ridiculous. It's like this shouldn't, be this hard.

B

Yeah, well, I, like I just given up like I just let linux do whatever it's saying.

C

B

My monitor one is two monitor. Two is three and I don't know it's.

A

Yeah, I just keep one monitor on my linux box. There's a reason for that: multiple desktops, but one monitor.

B

One monitor yeah, maybe.

A

B

Should do that here, um so you got you guys, you see the presentation right these, yes, am I presenting the right thing, all right cool, um so I'm gonna kind of talk about operators from an overview right. So I know you guys hopped on to talk about get ups, but first I think we need to talk about operators right and um what operators are and what what they provide right. So um you know, if you've ever been to any of these streams or talked to anyone at red hat.

B

This shouldn't be new to you, because all we do is talk about operators right. So um the idea behind the operators right. It's like you, want a cloud-like experience everywhere right. So what's what is? um What does that actually mean right so um operator, and the operator framework is um uh a way to codify operational knowledge right.

B

We call it an operator, because the idea is operational knowledge we want to codify that right and and what is that you know what does that actually mean, and I think andrew right, by the way, if any of you um haven't caught andrew's shift for administrators office hours, I'll.

A

Highly recommend.

B

That, especially if you're an admin.

A

That's a great show as well. You need help.

B

Great show right so, um but I think I think andrew kind of described it. He described it on the show as well very perfectly right as an admin right or as as a devops person right. Let's take a simple use case. If you spin up a mysql database right, if you're, if you're doing that over and over and over and over again right, you essentially do the same steps all the time.

C

B

For your environment, right, like it's, not the same steps for everyone, but it's.

D

B

Specific for your environment, you perform the same steps over and over and over and over again right so um and a lot of the ways people have already codified that right. So we have ansible right. I think ansible has a very large ecosystem that.

A

Massive she built.

B

A company he actually built a company around that before red hat acquired them, um and so you know a lot of that stuff's already codified. So um we want to kind of bring that into kubernetes native way right. So we we've already codified, I think people say operators codified, operational knowledge. I think people have already done that and I think companies have already done that. I mean for a while now right, like even before ansible existed.

B

I did things with uh puppet and chef um to codify a lot of this operational knowledge right um and even before, then people would just fire off. uh I remember doing uh solaris administration right. It's just nothing, but a bunch of shell scripts running in ssh right. So, like we've already kind of done that right.

B

So we kind of took that idea and um put it in a kubernetes native way right and it really is to um deliver applications, uh automate workflows and uh to really um provide a sas like experience for end users right so.

B

So really it's you've, um you take the knowledge of you, know, operators and operators in that like people right so, like an actual, you know either an sre either. You know devops engineer admin guy. They know how to set something up right. You embed that knowledge. You take that knowledge and you put it in what we call an operator and an operator and I'll go over it. A little bit is a um kubernetes native way right. We took the principles of uh custom, resource definitions and uh put a controller behind it.

B

To actually do something right, so we embed the knowledge and operator and operator all it does. It manages, deploys and um automatically things like deployments staple sets, um secrets, config maps. Anything you want anything you want to be able to codify you. Can it's not an operator right um and so really what? um What are the capabilities right for in in the sdk right? So we have an sdk in order. You know um in order to easily build operators um right so like.

A

And this is a cncf project now, by the way.

B

Yes, by the way the the operator framework is actually in the sandbox, I believe they just got.

D

B

They got into sandbox, so they it's a cncf project now right.

D

B

So, like everything red hat, does we like to continuously donate things um to the open source community? um So we have uh helm right so like so, if you're already having helm, charts put that in an operator you're good to go um ansible right. So for you um uh for you admins right, so you admins hold a special place in my heart, because I was the former admin um nsre. So it's you know. I understand the pain, um your ansible scripts.

B

You know if you're already using ansible, you're gonna fit right in right, so I've written a ton, a ton of ansible uh operators right. So if I can do it, it's not hard right.

D

B

So don't think it's like, don't think it's hard right so like if I can yeah. If I you know, I just took out my ansible scripts. I put them in operator and you know minor tweaking and they just work right. So um so it's um very very extensible and then for anything else. You just have go laying right. So that's kind of like the catch-all, I would say.

B

I think um if you can do it and go you pretty much, do it um anywhere right so.

A

B

Like I said before, we.

A

Did uh we did a whole thing on this once didn't we like ants will.

B

Offer yeah um I.

A

B

I think yeah yeah, you probably have a streak. If we haven't done a stream about it, we probably need need to get the ansible team on it's time.

A

For operator from scratch, yeah, I got it right here. Yes,.

B

Yes, actually yeah me and uh jason. Did it right yeah we did a uh ansible operator from scratch. So.

A

And then I did a workshop several workshops now on.

B

Well, you were on the ansible team, so if anyone would know that.

A

Yeah, no, uh that was like that was part of the draw right. They knew they were working on answer operators. They knew, I knew a lot about kubernetes, so it was like we need to hire chris yeah.

B

Yeah exactly yeah, so we need that guy.

A

Yeah, like the venn diagram of ansible and kubernetes like world, was like only a handful of people at the time. A couple.

B

Yeah yeah a couple years ago, was like three people and you were one of the three.

B

It was like literally something like that. um So operators, essentially, like I said before, is nothing special right. So if you are in the kubernetes world cloud native world, um there's something called a custom resource definition right, so very high level right, so you know I don't want. I can. Actually you can actually talk the whole hour about just crds right, but um custom resource definitions essentially extends the kubernetes api right. So when you do something like cube, ctl get pods.

B

That pod is an actual object. You can actually extend that api, so you can do cube. Ctl get uh um barf right. Yes, we're talking about spaceballs earlier on the chat. um Why not the um you know the cube? Ctl get barf and- um and you can have the kubernetes api- understand that right or you can say you know same same thing as a pod. So that's what a custom resource is. um So you can do things like in this example. Get fcd cluster right.

B

um What an operator does is, um you know you can have a custom resource and a custom resource definition, but kubernetes doesn't actually do anything with it unless you have a controller right and that's what kind of like what the operators is. Is that an actual controller, I would say a controller on steroids right? um It watches the uh events. It does a reconciliation.

B

It does. You know xyz everything it needs to do whatever you codified in it right and then the operator then um modifies um and manages uh the kubernetes native resources right so again, deployments staple sets, auto scalers, um there's nothing, uh there's nothing, nothing special about them other than what you've written in the actual operator or what the isv um writes in there there's. You know we're trying to build an entire ecosystem, around um operators right and trying to get our isvs, and you know third-party vendors uh to be able to um write operators.

B

So that way you can use it in your openshift cluster right. So um so, with all that in mind right, um you need a way so like after a while, you have you're going to have a ton of operators you're going to have operators a couple of operators and couple of operators.

A

Meta operators, all kinds of operators, yeah yeah,.

B

You're going to have you're going to have operators depending on other operators, so um we need so this. So we have the idea of the operator lifecycle manager right, which is from a high level, an operator that manages operators right. So it's kind of like a a layer, a layer on top, so just from a a quick overview right.

B

The the operator lifecycle manager manages the lifecycle of your operator right, so you can say hey for my cluster. I want my operator to be pinned for this specific cluster. I want it to be pinned to version uh 1.3 right, so you can have it and then 1.3 the operator does whatever it does. um If there's an upgrade to that operator, you can choose to either upgrade it or stay on that version um and it's kind of like a cascading effect right.

B

So if you upgrade your operator to here in this example from 1.3 to 2.0, then your operator may upgrade to other components right, so the operator may upgrade the software or you know the version of my sql, whatever it's managing it'll upgrade it right, so um so yeah. So you have um so you have operators that do some sort of automation. You have the operator, lifecycle manager that automates the automation, um and so we also, um you know, have the operator hub right. So the operator hub, you know just kind of quick commercial here.

B

Is you know um away? You know, for you know it was launched by red hat for way for, like you know, um you know red hat aws, google, whatever uh to be able to publish their operators, so that way you can consume them. uh The idea is in any kubernetes cluster right, so it's not necessarily an openshift specific thing, but you know any kubernetes uh um cluster can use an operator, um and so the operator hub is integrated directly into the openshift console.

D

A

um If you were on the channel earlier, you actually uh witnessed or waited on me to install ocs earlier, which happened in a.

D

Matter of minutes and that.

B

Is a metaphor: yeah yeah.

A

B

uh I actually um glad you brought up the meta operator because, like my example, is gonna gonna be that meta operator sort of thing, so um so um so yeah. So now we have the operator hub. So then you, um actually you know you the royal you.

B

If you have an operator, um you can actually publish a community operator and have openshift users consume that right. So.

A

Yeah and maybe tweak it and make it better, yeah.

B

Yeah, maybe tweak it yeah, so I I think um um not not not to our own horn here, but you know this is a red hat channel. um If you you know, I think, like the quickest startup, the quickest startup idea could be like basically, oh, hey, um we're gonna, publish our application on the operator hub and now you're exposed the entire red hat customer base.

A

Right yeah, like a certified operator in operator hub, will get you access to every openshift customer yeah.

B

Every openshift customer ever so you know kind of just. If you ever think of an idea, it's probably a good for an operator, it's probably a good idea to put in the operator hub. So um this next slide, I think, is my absolute favorite slide ever that um I- and this is a slide- that one of our.

A

Oh yeah, I know it's far cry. I know yeah, you know what it's like. First of all,.

B

First of all, it makes you laugh when you first see.

D

B

And then it makes you think because you're like oh, that, doesn't make sense right, because that's what you're doing with an operator.

C

B

So kind of just in summary, right, um first of all, I just want to enjoy this. This.

A

Yeah, the gift is amazing here.

B

This is this my this is my favorite slide. I don't know it's just it'll always be my favorite slide. So far,.

D

B

So, operators, right so like you're, codifying operational software life cycle knowledge right, so you're essentially doing what this gif is doing right. This is like you're, putting all your knowledge into um it's kind of the evolution of I'm going to replace you with a small shell script right.

A

Right I mean now I'm going to replace.

B

I'll replace you now with an operator right right.

A

And you can actually write operators there's. I think it's called flank. It's a way to write operators with bash, so you can do that and and as someone mentioned in chat, you can write them with quercus and you know yeah the sky's. The limit, there's quite a bit of like cop, is out there kopf.

A

uh You can write them with python, so yeah like yeah, you sky, is kind of the limit as far as what you want.

B

To operate, I'm gonna check out that bash one, because that.

A

Yeah yeah, I can do.

B

A

Yeah game over right, like I can ride operators all day at that point, yeah yeah. I.

B

A

But ansible same thing right, like.

B

Yeah with ansible same thing, I've been doing it with ansible, just because.

C

B

Just what I don't, I didn't have to learn anything new.

A

B

A

B

Fit perfectly right so uh so operators, you codify operational knowledge, built on kubernetes, primitives right so cr days, um sdk golang, ansible helm as chris mentioned sky's. The limit currently right.

A

B

You know there's a sdk for anything now. um It's meant for automating applications right, so it's meant for automating, deploying application, stacks uh yeah, but openshift kind of just took that and ran yeah right see and the entire openshift four platform, if you haven't been paying attention, is built on operators, so even the platform itself installs itself and manages itself with operators right. So there's something.

A

B

The uh cluster operators, the ceo and the cvo right, the cluster version operator that.

A

B

The actual cluster itself is built on top of our operator, so we.

D

B

In so we don't just tell we we're not we're just dog fooding, we didn't just tell people to do it, we're doing it ourselves right.

D

Yeah we we done, did it.

B

We we don't, did it right like we're, not just telling you we're preaching right, we're preaching we're practicing, what we're preaching um and then to manage operators. You manage it with all olm right, so uh you don't have to right. You can obviously.

A

Install operators, you can totally have independent operators right. The idea.

B

Behind olm is to.

A

Manage your operators for you kind of deal.

B

Yeah yeah so like you can either manage in manually or you can manage them via olm, um so you can control versions and upgrade so you can have actually a little bit more control over your operators that way right and it's, and it gives you like that. Catalog sas experience right.

D

B

You know give me click, you know. Like the previous example, let me go back a couple slides. uh You know, I click a button and it gives me an xcd cluster right so like that's kind of like the sas um version of that right, so um so cool. So um if you want to find out more about operators uh where people go operator, hub.io operator.io somebody.

A

Is a great place to start. We have a book on how to write operators. I posted that that's just jason dobby's book yeah. So if you've seen jason on the channel, he helped write that book with josh wood who's also been on the channel before.

B

A

Yeah yeah, like I have it on my bookshelf right here, to give you any indication of how important it is: yeah I've act. I've actually referenced that book several times and and just trying to figure out. How should I do thing x, yeah, yeah,.

B

Yeah because sometimes you're like well, what's the best practice here, it's kind of good to reference, something right, you know, make sure you're on the right path, because I've always people always want to see if they're on the right path, at least right they're, not.

A

Yeah, like people want to make.

B

Sure they're not spinning.

A

Their wheels and wasting their time right.

B

Yeah yeah exactly well, people want to innovate, but they don't want to be too crazy right. It's like.

A

They want wanted to be maintainable, yes, yeah yeah, exactly.

B

Yeah um so um so um get ups and operators right. So it's like I like to say I always like to make now it's just like peanut butter and chocolate right, so like two things that you don't think should go together, but actually work together, pretty well um and so uh really, like. I said before, there's just like so many automation tools right so where um you know automation's, like everywhere, automation's everywhere, there's if you on the cncf diagram. If you see that poster of the landscape.

D

C

A

Yeah, no, no! No! No! I had it. I had I printed it out for my interview with ashesh. There.

B

A

If you've seen this before, you know what we're talking about yeah, yeah, yeah.

B

Yeah, it seems like everyone's trying to sell you some sort of automation right so there's automation in the face everywhere. um So um I I, when I was first kind of dealing with get ups, um exploring this this this this buzzword right, um I kind of just broke it down. It's like what are you know like: where does it all fit like? What are operators really right so like? Let's just break it down right? So an operator um is nothing but a crd right. So the custom resource is a saved in yaml right.

B

You submit a yaml to um kubernetes cluster right. The automation is triggered by the ammo right so by the cr right. I can either so I have an operator that the pods are managed by the operator right. So even if you scale the deployment, my operator will go. No, no! um You know I control the scale right so like so all the automation is triggered by the ammo um and, like I said, right, changes are triggered by changing the ammo. um Other changes are reconciled right so like.

B

If you make a change outside the operator, the operator will reconcile that say no. You need to change everything um with me so, like that's kind of like some of the issues I had with openshift four coming from three to four, it's like well, I modified the deployment. Why did I do this? It's like no everything's an operator. Don't.

A

Right, yeah, don't don't don't modify the employment modified.

B

A

Deployment is actually an artifact of the operator right like that's how.

B

You yeah exactly yeah, it's yeah, it's a hierarchically. It's it's it's! You know you're going. You have to go above right, um but one of the cool things I didn't mention in the other slides, but um with operators you can easily replicate. You know. If you have the same cr once you install the operator, you can easily replicate that to many clusters, um so you can kind of like rubber stamp. These things.

A

C

B

So then you know: how does that fit together? So what's get up so like manifests, are saved and get in the yamaha correct right so like we do everything yaml um or json, but you know I'm a mo guy.

D

Yeah, so the automation.

B

And the sig yeah the automation, the sync tools read the manifest, which is the ammo uh changes are made by changing the ammo and pr in the ammo right. So um as as you see like there's a lot of similarities between how you manage an operator and how you would manage uh get ops right. So um so so you interface with um the operator framework or just like the operators in general uh via yaml right. So um it's just yaml at the end of the day. Right so.

A

Yeah, it's it's all yaml.

B

All the way down, as they say,.

A

B

Yeah, it's yeah, it's yam all the way, yeah exactly so. If, um if you're operators, it's just a bunch of yaml right, I think that fits perfectly in with get ops because you're just storing that gamma right. So um so so the question is: it's not if it operators um fit into the gitops workflow, because it does because it's just yaml right, but where is um where's the point of demarcation right so that that's that's the thing right so should you save all the manifests that um your operator creates like? Should you save the deployment?

B

Should you save the secret? Should you save or should you let the operator handle that right and um I think that's. The most important thing is like not if operators and git ops works together because they do is like. Where is your point of demarcation right, so um operators are animal driven and the yama for the operator uh should live and get right, and I think that's where the um the point of demarcation is. Is this yaml right?

B

So you have a yaml for the operator that it's going to live and get, and you pass that over to the operator and then you let the operator do whatever it needs to do um with that yaml, whether you know the um you know, there's a there's a lot of um there's a lot of ideas around okay, like when you deploy a kubernetes cluster.

B

The first thing you do: is you export all your yaml, um you put it in git and that's how you manage your cluster um if you're using openshift or an operator driven uh cluster that changes right because you're not gonna export every single, um every single yaml file right every single configuration you're not going to um like you said chris. The deployment is now an artifact of the operator, so you're not going to save the deployment you're going to save the artifact you're going to save the the operator configuration um and.

A

B

The higher level thing right, so it's it's the point of demarcation, so I had a long long time ago, right at um hashtag last job right um at last, job um that was yeah yeah, exactly yeah. That's what I'm saying so, I'm reaching back right.

C

B

Where's your way back, um we had kind of a workflow kind of like a similar automation where- um and I think my ex boss described it pretty well and I think it fits here is like you have a pipe right.

B

You have the main line and you have where the water comes out of the faucet right, so you can either have this turned on and this turned off, or this turned off in this, like you need to um um you control the flow at a different point right. So, if you're going to turn on the water, you have to leave this open.

B

If you're going to turn on the water, where you are, you have to leave this open or, if you're going to turn on the water here, you need to leave this open, so it's um depends where you want to control the water right, the water flow. So um I hope that analogy came out um correctly. If my experience.

A

It didn't come out in a way that made me cringe, so that worked.

B

Okay, the answer is watching. I apologize if I butchered that, but it's um it's really where the point of demarcation is. The point I want to get across is where you want to. You know: have that demarcation right. So.

A

So frank asked a question. I think it's sure.

D

A

Time to stop and ask it to you: uh must a developer specify each used crd version, like group version kind of the created objects? Whoa? Oh, let's just move down to decide on which kubernetes version the operator can run on or how is the supported cluster version managed. So that's a very good question.

B

A

Matters right, like.

B

Yeah yeah it does yeah yeah it I mean for the cluster. Definitely it matters especially.

A

B

They're doing that um that alpha thing, because.

A

Right like if you're, if you're, using like what is it 1 15 you're you've got an alpha crd. If you're using 116 version of kubernetes, you have the the the ga version of the crds so or crs. So you can. You can yeah like that matters right like if you're on an older version of kubernetes, you kind of have to be careful with that right.

A

So that's where, like I like, to use a little bit of ansible to figure that part out right, like hey, tell me what version I'm on okay now deploy the the operator kind of deal.

B

Yeah, what and actually this is um you know this is kind of it. You brought up an interesting use case chris and by the way, thank you for asking that question, because it kind of um percolates, right kind of makes you think right where you can have with a combination of um you know, get ops practices, obviously, but to customize right. So where you can have like the template and customize you have the ansible um operator.

B

You can actually have like a generic version of your template deployed in many versions of your clusters. Now um I think we only support n minus one, and just like in general, like um just like in the community kubernetes is n minus one right, n, n, minus or n minus two. um uh I.

A

Think it's n minus two. I.

B

A

B

Minus two you're right now that I think about it.

A

Yeah because it's the current version, anyways yeah.

B

Yeah, if you're running 1.15, you should probably upgrade so.

A

C

All I'm saying.

D

B

But yeah you could have, you could have multiple versions of uh of kubernetes um and you do need to look out for that. uh Crd version right, like uh the the well yeah, what they call the the gsv right uh or the.

A

The group version kind yeah.

B

Group version yeah, yeah zvk. There we go um so yeah, so you could have using templating using um uh you know like customize and using ansible. You can kind of, like you said, figure out, okay. What class cluster version am I on I'm going to start patching some of these so that way, like ingress, just graduated, to v1 right.

A

Right, yeah right! So, if you're using.

B

Beta ingress you're going to get a lot of warnings right now. I don't know if you guys realize that you get a lot of warnings once you deploy you're like oh, this graduated to v1, so good question.

A

Yeah no great question.

B

Thank you frank.

A

For asking it cool.

B

Thank you um so one more. This is the last slide right and then we'll go into kind of just a little little hacking on the command line here right. um Other things to consider right so since openshift is built on operators, there's just like other things. Now you need to consider right um so, like, like, I was saying before cluster operators in the cvo right, so you have to kind of think about where um you know where, where is that sword and where's that demarcation line right?

B

So I some of that's not clear to me yet just yet right so by the way, a lot of the stuff on this list is not clear to me yet right. So we're not going to talk about this. This will be the later show once again.

A

Oh get a chance to.

B

Yeah, so this is kind of like a teaser right um machine, config operators right. So, if you've worked with openshift, you know that we have a machine, config and machine config operators right so um and machine sets right. So like we have a red hat core os machine set windows containers is coming by the way. um If you guys don't know, not only am I the get ops guy on our team, I am also.

D

B

Containers guy so maybe I'll do like a a joint show about windows, containers and get ups, and that's going to be kind of cool. That.

A

B

Yeah now that'll yeah it'll ga next week we're targeting.

D

B

Week or the week after, I think it's the week after um I forget.

A

But jp dade, your thing is coming.

D

B

It's coming way sooner than you think, yeah it's coming soon.

A

Machine configs.

B

Right so machine configs for those who don't know is um when openshift boots up a node. All this configuration is actually stored in ncd and openshift applies those via machine configs.

B

As you know, uh machine configs, you can have like a hundreds of them, so we have something called a machine, config pool, which is a aggregate of all those machine. Configs like where are those stored right? Do you get up to those um there's just so much to consider in an operator world where that demarcation is it kind of gets a little hairy? It kind of gets little there's a venn diagram, multi.

D

B

Dimensional ven venn diagram right. Is that what I would like to say um so yeah so uh like I promised that was my last slide. um It's uh any any questions in the chat that we can get to before. I start hacking.

A

So I kind of answered this, but you know frank, also asks so. The operator framework cannot check if a special operator version can run on the kubernetes cluster. We have to keep the operator always up to date to run on old as well as new clusters. So this is an interesting question again frank. Thank you for it.

B

A

What I do is I you know what, if I'm running an operator, I want to know what version of kubernetes I'm on so I know which kind of crd or cr to call which version of it. So I always have a little script that just says or a little playbook.

A

Essentially that asks that question and registers that for me, so I know to deploy to version a b or c right like because I know there's only a handful of api versions, I'm going to be able to deploy to at any given point in my environment right now, it's one.

A

So if I'm writing that operator, I'm going to write some logic to do like a quick oc or a quick. You know call in where the ansible to grab that version of kubernetes so that I know which api to use and then I can have those variables in my uh operator later on.

B

What's what's really cool about the operator sdk is that the downward api is available to you, so you can just start querying the um the kubernetes api really really easily. um I do it uh probably not in depth as you can do it, but there's like this whole. You got the once you get the the sdk downloaded and loaded into your cluster. You have access to the kubernetes api, so you can do a lot of fancy things um and I think a lot of it depends on your ci cd process right.

B

um One of the things I've been threatening to do is talk about the ci cd practices on this um because get ops, you can't have get ups without ci, cd, right and.

D

C

Been kind of you kind of.

A

B

Of been pushing that topic off a little bit a.

A

Little understandably, because.

B

So understand that yeah it's.

A

A lot of nuance to it right, like you, can't just.

B

There's a lot a lot of nuances.

A

You can't just go out there and have I think, one episode. It's gonna be like an episode and a half. Maybe two right like before.

B

You go to gonna, be like I think it's gonna be like a series right.

A

Okay, fair enough.

B

So uh uh jp dade has said um talking about sent four. So yes, I'm.

A

Wearing the scent.

B

Force one t-shirt right because I am, uh we should have coordinated our t-shirts.

A

I know right, you should have told me what you were wearing yeah.

B

Yeah, um because as an admin right, like I'm, a c linux guy, what's really one of my favorite things about openshift that um I don't think we talk about enough. We kind of blanket security over it, but really um yeah that wash right now.

A

Yes, absolutely love it.

B

As yes, because we have the sea legs, one of my favorite things about openshift is that I don't have to think about sc linux right. That is abstracted away from me. It does all the sc linux stuff for me, um whereas in the past, as an admin actually had to know a lot of se linux, I don't know if any of you took the rhce tests version five version, six there's a lot of messy learning stuff on there and it's like yeah yeah, that's the exact face. I made.

A

Yeah, exactly just because.

B

It's not that it's hard, it's just like something else. You need to think about, and just another nuance right where that now that has abstracted um abstracted from you know, openshift, and I think that's like one of my favorite things about openshift that we don't talk about enough. I don't think is that openshift abstracts the need for you to know sc linux, so um your developer doesn't need to even worry about it. You don't even need to worry about it. Sc linux, when you're running openshift, so um it's kind of abstracted for me.

B

So that's yeah kind of um that's pretty cool. So um yeah stop stop disabling us linux.com! Absolutely um we do not.

A

Gonna make dan walsh cry and no one likes to hear him cry.

B

No one likes to hear down walsh. Yeah cries in uh boston right. I don't know.

A

Yeah I mean if you hear something from the northeast direction: stan walsh crying about seven four, zero yeah.

B

So uh make this a little bigger, uh so I have a cluster here, uh cube ctl and this is a fresh cluster uh get nodes. um Is it still up all right, cool.

D

B

Actually, one of my favorite things is a sort buy.

B

I do sort by workers, but I don't know why it becomes last in the list but anyway, so I have three masters three workers I just installed this this morning. um So what I'm gonna do is, let's first install argo. So by the way I haven't tested this on um yeah yeah, so.

D

B

This might not work so I apologize. It's been one of those weeks where it's just been crazy. Back-To-Back meetings. Chris, I don't know, I know you. I know you know so I don't have to complain to you right. It's just back-to-back stuff. I haven't been able to test this on 4.6 if it works on 4.5. So if it doesn't work, I apologize so we'll debug on the fly here. So let's install argo first.

B

One of the things that um the canadians right, the the get ups canadians at red hive say that.

A

Our wonderful red hat, canadian.

B

Friends, yes, yeah yeah, there's uh a lot of the new get ops. Language is shell, because you just basically do.

A

Well, let's think about this, for a second.

B

Do an until right, you do an until apply and it won't return until it's successful, I mean.

A

Right but remember a lot of kubernetes the management of it, the installation of it all that fun stuff was written in bash at the beginning right. So it's like.

B

That's right, redoing.

A

That cycle, so you know the get ops. Maturity will start showing here as soon as you know, more languages start getting involved.

B

Yeah yeah, so let's see here come on, go, go, go, go, go all.

A

D

A

Unchanged, configured unchanged no matches for okay, so I was waiting.

B

A

We go all right.

B

Yeah yeah, that, which is why? What the until is.

D

B

Let's do this watch oc, get uh you pods.

A

Spell I believe in you.

B

All right so that'll fire running, that's running cool, um so that's and I'm going to deploy this repo. It takes a while for this repo to deploy so I'll, deploy it first and then I'll talk about what's going on here.

B

Go baby go baby, no whammies.

A

Great a great yeah.

B

Right cool: this is.

A

Doing a hormone good job, buddy.

B

All right swing all right, so let's uh do octotree by the way, whoever someone on the channel.

A

Finally, finally, by the way.

B

You know what it's been a lifesaver. To be honest with you.

A

Yes, so thank you, I don't know who? Yes, it was me and narendev there.

B

We go, yeah miranda was like.

A

Christian, you need octotree, and I was like christian go to octotree.io sign up done.

B

So uh so yeah, so I have to keep clicking around right, so I can see.

C

B

So I have um so, as you know, argos td uses a concept of applications, and um so, if I could go over here, if I look at um I'm going to be installing the efk stack right, so the fk stack is um log.

A

B

And it's actually yeah elastic, search, elasticsearch.

A

Fluency cabana yeah.

B

A

B

Like, like, like you said, it's a meta operator, so um so it's one of those like you need like three operators.

A

Right, like you, gotta, install all three of those programs connect them together. That's what a meta operator is for.

B

So I have this efk application that basically says um it's. This is argo cd application. It says, uh go to the manifest efk install. So let's go here manifest efk install.

A

Using the f out of that octo tree.

B

Yeah I love this option.

B

um Can I go directly to the directory that can I click.

A

I think if you click the directory name.

B

D

Quite figured that out apparently now, if I could go.

B

To the directory, but.

A

B

A

B

A

Inside and then go up a level, which is what you did so yeah.

B

Yeah yeah, that's that I always do it that way. So by the way, if you guys know, please put it in chat, um so I have here um the the cluster logging operator right. I have this instance, um which is basically it says: I'm gonna use a storage cast gpt2, so um it says to use base right so in this base directory.

B

I have all the operator configurations that I need right. So in order to get an operator up and running, you need an operator group a subscription and the actual cr.

C

B

The operator right so um and there's this implicit namespace right so like you, need it in an ancient space, so um I didn't mention it because you need a namespace for everything nowadays right. So I.

D

B

um um You know openshift logging right, which is the namespace I want to create. uh I need another namespace for the um to the actual operator for the actual operator to live there, um but the actual magic happens is first in the operator group right. So I need an elasticsearch operator group, so that's that here right so I say all right, so I'm going to create operator group um uh for elasticsearch and I need an operator group for there's another one for the cluster logging um right. So it basically says for this operator.

B

I want to target this namespace um and then once I have that, then I need to subscribe to the operator right. So then I have a subscription right so subscription just says. I want to subscribe to this particular version of the operator right because your operator may have different versions, so you need.

A

To which goes back to the gvka thing right, like version 3., one can handle older clusters. 3.2 can handle current clusters. 3.3 can be ready for the next release of cluster kind of thing.

B

Yeah exactly right, so then you have so then that could be so your channel. That could be something you either do it ansible or if you do it customize, you can do it. You know wherever right, but that's basically, where you say I want to subscribe to this particular version right um and uh oh yeah. I will share my repo link.

A

Yeah I was trying to look it up, but yeah. Thank you for seeing that.

B

Yeah yeah, um and so uh so this gives me so basically, as you can see here, um I don't have anything specific like uh well. Let me go back to base. I don't have anything.

B

I wish I can click on the directory anyways um I'll treat people. I wish I can click on directory, um so we'll.

A

See if they're watching.

D

Yeah see if they're watching okay.

B

So we don't know right um so yeah, so I have. um I don't, have anything in here specific to um the actual deployments. So, if notice here, everything is either an operator group a subscription um or the namespace right. There's there's nothing defining like oh hey. uh I need I'm saving my deployments and my deploy or my deployment configs or my stateful sets, or my storage or yada yada yada right.

D

B

All I'm saving is the operator manifest and then I'm gonna, let the operator handle, do its thing right. So nice.

A

This is a good thing, so.

B

Yeah yeah yeah, so so my point of demarcation is the operator right and operator hub, I'm going to just submit the manifest and then the operator just does its magic. So all right, hopefully we'll see.

A

So I just found a cool feature in octotree, not to hype it up anymore, but it actually has dark. It has dark mode for all of github as a feature.

B

A

B

That's why yeah? I was actually.

D

Looking at that, I was looking at because.

B

You know I like it so much that I was just thinking about just like just buying their one of their tiers.

A

Yeah they're, um I have fifty.

B

Because remaining and.

A

This is definitely probably gonna, be an expense that I do.

B

Yeah yeah I mean, and it's like it's like cheap. It's like it's like 30 bucks a year or something like.

C

B

um Anytime, I like something I always buy, or definitely.

A

You're, like you, gotta support, crypt.

B

I yeah, let's encrypt- I always donate to let's encrypt just because I just use a crap out of that.

A

Yeah I donate to eff every month, so yeah.

B

The uh so yeah.

A

B

A

How do I yeah okay, there we go.

B

Yeah now I want to register then now I'm gonna go into argo. So if this works, this is the first time if it works is the first time I see it work so.

A

That's scary, I don't know what that is.

B

Well, I haven't had time to test it, but it should it should work. uh You know.

D

B

Yeah, so I got the efk well, it's.

A

B

Sync, let me try to sync it: um let's just prune it, let's just manually sync it another operation is in progress. Oh, I probably have to cancel it.

B

A

Jp data is not so much an expense account. It's just that if there's tools that make our job easier, we're encouraged to uh not give money to red hat. We're encouraged to get that money back from.

B

Right, yeah, yeah.

B

Let's go back here, so I might have a error here. I have it's missing: okay, yeah! This is uh so by the way- um and this is something um I've been trying to play with. If you saw here. This is something I might talk about in in for their episodes. I have um a sink waves right because there's like certain things that need to happen in certain orders um in order for something to work, that's just life right. It's just right, just life in I.t in general.

B

Right, you need to do abc before you do d um and that's what sync waves are and triggers, and things like that and I've been playing around with that, um but looks like. Let me just delete this resource here.

B

A

Joking uh remember back in the old days yeah I'm just going to delete this server. People would freak out.

D

No, you don't know.

B

What that's connected to.

B

Yeah I have. I have uh um yeah, there's there's like funny stories out there, where it's like someone unplugged an xbox and the webster website went down. I was like what.

A

Right yeah you're, like.

D

C

A

Yeah I've got some pretty funny stories from those three days: people accidentally unplugging things or wiring things incorrectly. Potentially, you know.

A

Wow it actually lets you pick fonts and everything. Well, that's cool! Sorry! I'm still talking about up! Oh.

B

Yeah, it's just playing with octo yeah octatree here.

B

Yeah yeah: this is seriously messed up here. This is why I probably should have tested it.

B

Yeah I'm trying to sync the individual.

B

It might be because.

B

You have the right storage, um oh.

D

A

Are you just spinning or ssd which, which is your cluster, seeing them.

B

Well, I'm on aws, so.

A

B

uh Storage class gpt2- okay, that's that.

A

B

That should be that should work. Yeah sync. Can I sync this guy.

B

Here, where do I see the logs here? Let's do this, let's just just debug! This live.

B

Here was my commit, so let's see here, let's see what it looks like. uh Oh, I know what I think I did hold on.

B

B

Okay, that's fine gpt2!

B

That is silo instance. No, I do have it okay, so I have this: skip dry run on missing resource.

A

A

B

Eve naughty me: oh wait. How do I go back now? I'm stuck! uh Oh here we go here. We go again there we go.

A

D

A

Bought a license to octo tree.

B

Yeah I mean you should support the things we like to use right.

A

Yeah, I agree: sync them apps, uh which one uh yeah.

C

A

C

That is a problem.

A

um Which is it? Which component.

B

It's the actual instance of login.

A

D

A

B

It's the actual cr right.

A

B

uh The login component that.

A

B

That doesn't matter right. Everything else is good, doesn't look like, so I would think it stops syncing. Let's terminate that.

A

Braided silver: this is just as bad as the time his github get ignored.

D

A

Remember folks, this channel is designed for all of us to learn together right, like we embrace failure on this channel, because if we don't, if we don't fail and learn from it, like nothing's, perfect in life right, so we want to make sure we give you a very real experience, and this is going to show you how to troubleshoot problems later um we don't, we do it live. These aren't polished demos for a very good reason.

B

So, let's sync that.

A

Okay, let's see what happens here well,.

B

It's funny everything else is.

A

B

A

Sync, you've been drinking a lot of.

B

B

Yeah, it's just this guy partial synchronization, not recorded in history yeah. Well, I know that stop syncing. How about that you know. What's funny it doesn't. Let me like, because I have auto sync turned on when I stop it. It'll like it like it, kicks back on.

A

Oh yeah, that's right.

B

Oh wait: I can do this. Okay, yeah see, look it just like it kicked off again like I don't want to. I want to debug this damn it.

A

Let me debug you.

B

Well, what's what's funny um uh braided silver.

A

B

I I I know um I was I was talking to her about like because I've been doing a lot of goal line lately and I was.

A

Like you know, that's okay, we're doing.

B

Golang well doing golang, but, like I came from like the web front, is doing I going from php. I went to golang and she's and she's like that's like going from being a fish to being a bird. I'm like get that pretty.

A

Much yeah, like I, don't think anyone.

B

Has ever put it quite so eloquently as as.

A

That's very well said: yes,.

B

Yeah, so let me uh let me actually, I'm gonna cheat a little bit guys. Sorry, uh because I need to see.

A

Braided silver ass: can you write in a weight somewhere, even if it does turn it back on? It's like a sleeper or something.

B

Yeah, so that's what this these sync waves are so like. If I have um oops where's shift be yeah, I have a sink wave. You can do like you know. It starts at negative 20 and goes to plus 20 right. So it's kind of like a um uh it's kind of like the like.

A

A golf course right: oh okay, yeah.

B

Yeah yeah, so um so yeah so like zero is actually the highest number I put on because I have it like negative five negative four, so it should just work. So, let's see here, uh I'm gonna cheat a little bit here, tmp just to see. If I can get this going, I need to figure this out why this is a problem on 4.6.

B

Okay, so that's the problem, something.

A

No matches are.

B

Probably they're logging.

A

What are they, what are they changing.

B

Yeah, that's! That's! That's exactly that's exactly my question, so I guess um I don't know who asked the question but like this is now that that's that same issue just now biting me in the butt. Oh.

B

So as eric jacob says when in doubt.

A

Read the docs yeah yeah. Aren't you the one with the fancy gcp shortcut? Didn't you tell us about.

B

That yeah, um I do ocp space uh logging.

A

Oh sorry, no ocp, not gcp gosh.

B

Well, yeah yeah! No thanks! Thanks for reminding me um no.

A

No problem, I use it all the time. Thank you for telling me.

B

So cluster logging logging openshift, I o v1. That should be fine.

D

A

Hang on, let me see, I've got a 46.

B

B

I wonder if I could do.

B

This I have no idea what that.

D

B

Is uh let's go back to this guy here.

B

I need the whole http guy.

B

Yeah, so even customizes.

B

A

Open shift monitoring projects.

B

hmm It doesn't even like my.

B

Looks like this doesn't work on 4.6. I should probably.

A

Looking at the 4.6 stuff right now,.

B

Well, I wonder if I oh.

A

Well, the name space is open shift monitoring, I'm not sure if that changed from four five to four six, though I don't think it did.

B

So scrutiny still and start over here, oh.

A

Cluster monitor yeah.

B

B

So this is yeah. I need to go.

A

B

And see why this isn't working.

A

Oh I've got that trick shape you date hang on buddy. I got you so you add a search engine in google settings.

B

It's it's actually really cool yeah. It.

A

Is so I just have to.

B

A

B

And then I just type whatever I want yeah.

A

So you, if you put in.

D

A

If you go to your browser's setting for search engines- and you say manage them, what you basically do is you know you create? You add a search engine, and um let me edit this, so you give it a short code of whatever you want. You know, open shift or ocp is what we use, and this is the string that I actually use in that field. So that way, every version that gets updated. You just change the number and you got the latest docs that you're searching.

D

A

Type ocp so add it as a special search engine. It will save you tons of time but make sure that the version number is the version number the cluster you're using right now.

B

Yes, yeah um and that's that last little bit right there 4.6, so I always yeah like so every release. I just change it.

B

I think I got a race condition.

C

B

Yeah, I need to figure this out here.

A

Does anybody have any get ops questions for us that are yeah not related to what's on screen or any openshift question for that matter?.

B

Anyhow, we always have a question that I'm at that's right.

B

C

B

I think I really messed this up.

B

Let's see here so oopsie get applications.

B

Let's see delete applications.

B

So at least give me.

B

D

Time but I think I'm gonna.

B

Be running into yeah. Well, I think I'm getting into finalizer issues. Oh.

A

B

Yeah, so this is well. This is something that um also, I think I may switch to one of the one of the red hat canadians.

D

B

A job to actually trigger the efk deployment, instead of actually giving the cr.

A

That makes sense.

B

A

Mikey, a job.

B

A

B

In order to make sure the cr crd is in place before he deported.

A

Operations thing yeah.

B

Yeah yeah, but he uses a job which is which is nothing more than a shell script, which is why we exactly which is, um which is why we yeah we always joke that bash is the new, a git ops language um that basically just loops and makes sure once the cr is available, then he then he continues on with the um deployment here, but this isn't deleting. I wonder why I probably have storage issues is what I'm thinking, because if I do oc get pv, there's no persistent volumes, we'll see you get pvc there's!

B

No persistent volume claims either um and it needs uh storage. Oh.

A

B

Not my answer, no worries.

B

uh There is a way there's like a weight equals no.

B

B

Oh there's alexa.

A

Something went wrong, but everything worked just fine.

A

That's weird jp jpd says that string isn't working for him. That's weird, so you have to add it as a new search engine by the way.

A

And I apologize for the dog.

B

Yeah, I got some uh some dangling issues here. I think I may also have our back issues. Let's go over here.

B

Let's sync, this guy here.

B

Yeah, the reason I think I have our back issues is because when I go here- and I delete like this guy here,.

B

B

I cannot delete resource as the user that I am so you may have r back issues as.

C

B

So will this even delete? We don't know.

A

B

Yeah, it's fun, oc, get events, argo cd.

B

C

A

Carlos santana from our colleague on the ibm side, we have been trying to create a tecton pipeline with argo cd for operator and olm, bundles catalogs, interesting.

B

I hit a blocker running opm yeah, so like opm, so for those who don't know is opm is a way to bundle your operators right because you have an operator so what an operator when you're building an olm into olm, you need to build like all the metadata right so like when someone subscribes to version one of my operator. It goes out and downloads or gets the this particular tag of this particular image.

B

Right so um and you can bundle different versions right, so you could have one one bundle with different versions depending on what you subscribe to. You get a different version of the image right. So um so that's what that's? What the question is he's talking about right now about the opm bundle um in tecton?

B

So I guess what you're doing is so we need to see any folks push code, okay, so you're. So I guess mr santana here big fan of your music. By the way um I bet he never hears that joke. um Never it tries to auto, tries to automate. So when, when you, when you push the code, the code like, I guess, compiles, you create the operator and then you're just trying to automate adding that bundle into the new bundle into the operator. I guess is that what uh what you're doing.

B

A

So it's like, you need a whole versioning system to then plug into the catalog.

A

A

Carlos I need to hook you up with some of my friends. Send me an email, buddy yeah I can. I can get some people that are real smart on that, for you.

B

Sso you want uh dash dash insecure.

B

You want this guy, I'm trying to use the argo for my last ditch effort, I'm using the argo cli.

A

D

A

Stop in 20 minutes, that's all.

B

Oh no worries, I know we're even over time now.

A

That's fine people, like your show, so that's okay,.

B

Yeah I'm like langdon, show I'm just kidding.

A

B

B

B

Yeah yeah, the sun doesn't even say yeah. It's the wine um uh app list. Okay,.

A

So braided silver asked: is this a fresh cluster? If not did you stop any other running operators to make sure nothing was conflicting.

B

Oh yeah, it was a fresh cluster. um I could do by the way you can do oc get co and you get the cluster operators.

B

um And it tells you the uh the status, so I I've always this is such a programmer or engine, not even a programmer engineer mindset.

A

B

You see false, but false means good. In this case, nice.

A

Super helpful thank y'all.

B

Yeah, in my mind that this seems bad, but the degraded status is false, so um maybe they should put it like. I mean regular people. Non-Engineers would want you to see no there instead of false, but.

C

B

Yeah, no degraded, no right so like that would make more sense to me, but um but anyways. None of these operators are degraded.

A

We're not here to make sense of things.

B

Yeah, I know we're just here: uh progressing goes false. That means there's no operators progressing and you know available equals true, so um so yeah. So let's do uh argos, cds, ergo cd, uh app get- and this is here, tells me yeah. So it tells me this so there's there's some issue with uh order. Order of operation looks like um it doesn't want to delete it for whatever reason.

B

Yeah, for whatever reason it doesn't want to delete it.

B

And I believe, we'll see uh project uh open shift, oops, let's look at the actual openshift blogging. Oc get even also get pods and see here, nothing, okay, good! Oh so you get uh events.

B

Nothing well that that sucks.

A

Well, at least you're, honest.

B

D

B

Something uh defaults: let's do oc delete.

B

D

B

Just gonna drop the hammer for my.

A

B

Let's see what happens I'll see, projects.

B

Unlocking not there.

C

B

Come on finalizers, okay, so that's missing: let's delete the operator group.

B

Probably don't want to do that I'll really mess up. My cluster.

B

So subscriptions let's go see, get subscription.

B

A

What is your github username christian h, eight.

B

Christian h814 yeah, okay, see here alpha.

C

B

I just realized my my error error in my ways, so.

B

Yeah, so I subscribe to the 4.5 channel instead of the 4.6 channel. ah um That's why I may be having trouble so what so? Okay! So let me back up and I'll I'll explain what's happening right, um which we probably won't see this work within the next. You know couple minutes that we have left here.

A

B

um But what end up happening so when you subscribe to a channel and it's an older channel um and and you and there's a newer version and so like all these things like, I had like this perfect storm that just happened and you have your um uh and actually andrew pitt. He. He warned me about this too, so I'm just gonna kind of like lang didn't warn me about the github thing. uh He warned me about this uh error coming up here. So let's, um let's take a look here. um Where is.

B

It uh it's in one of these. Let me see if I can find it real quick um so like you can have your your operator automatically upgrade right, so you can have um your um there. We go. Install um approval equals automatic right. So what ends up happening is that.

B

What ends up happening is that I install 4.5. I have upgrades set to automatic, so then it um once it fully installs. Then it upgrades itself to the latest version right. So right.

D

B

What what's happening here- and I messed this up so bad- that it's not gonna reconcile right, like I have to like delete everything and start over.

D

B

Is what would end up happening is that um um I uh I installed 4.5. It saw okay I'll solve 4.5. Oh, you want by the way there's 4.6 available. So I'm going to upgrade you to 4.6.

D

B

So it was waiting that time that upgrade time right, so it and it went ahead and installed the entire operator right and then tried to upgrade itself, um and so I ended up with the race condition. So anyway,.

A

B

So that sucked so but.

A

At least you know.

B

Why there you go well, at least, I know why right yeah, yeah.

A

Sometimes we have to tweet out what broke after the fact that.

B

That's happened before actually.

A

Now we don't have to it's in the recording.

D

D

A

B

A

Yeah, you know maybe next time you just do a quick run through with it working yeah at the beginning. We touch.

B

The next yeah, exactly so cool.

A

B

So yeah so yeah.

A

No great show uh there's an sop for this: I'm almost certain uh yeah carlos santana. We we all need to talk. I feel like with what you're trying to do um so I nudged uh scott mccarty and dan in the github issue you opened so we'll get that figured out anything else, christian other than thank you audience. We appreciate you tuning in.

B

Yeah yeah, thank you um appreciate you tuning in. um I guess you can. uh You can tweet at me you tweet at christianh814 or find me on github it's the same name. I use the same name across everything, so um so yeah. So.

A

Yeah carlos santana.

B

Wants to know, if you have any experience with the argo roll allowance, so yeah argo rollouts is something I want to. I want to cover in future episodes, because I think it's um it'll be helpful, um yeah totally too yeah. So awesome.

A

Cool well good, talking to you. uh I hope you all stay safe out there and we will catch you next time.

B

Yep definitely all right, bye.

C

C