Red Hat OpenShift Ask an OpenShift Admin | Red Hat Livestreaming, 25 Jan 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ask an OpenShift Admin (E92) | Data driven fleet updates

Description

Join us this week as we welcome Scott Dodson to the livestream to discuss the "behind the scenes" processes for how we create, test, and ship OpenShift update and upgrade paths!

Want to learn more?

Try Red Hat OpenShift: https://red.ht/3uVxkya

Follow us:
Andrew Sullivan https://twitter.com/practicalandrew
Jonathan Rickard https://twitter.com/jrocktx1
OpenShift https://twitter.com/openshift

Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_c...

About Ask an OpenShift Admin:
Do you have questions? We want to answer them! Every Wednesday at 11am ET, hosts Andrew Sullivan and Jonathan Rickard sit down to answer your questions about Red Hat OpenShift.

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3rYz5ZS
OpenShift on GitHub https://github.com/openshift/

#RedHat #OpenShift #Kubernetes #ContainerPlatform

A

Live stream, uh this is this, isn't normal Johnny? What's what's happening, I I? Don't everything.

B

Man, everything we got to flip the script today.

A

Yeah today's uh today's a little different one, our interface here in uh restream, is broken. We didn't get to do our normal intro, video and all that other stuff. You know I gotta, mentally prepare for these things um and two uh Our Guest today. Scott has very graciously at the very last minute, uh bailed us out uh so Johnny and I had a topic had a guest. They had a conflict at the last minute and I Scott and I happen to be interacting in the same thread yesterday and it it clicked in my head.

A

It's like oh man, it'd, be great. If, if Scott could help us out and very he was very graciously agreed, but we only have him for the first 30 minutes, so basically till 11, 30 Eastern, uh so I, I'm, Johnny and I are going to skip all of our normal stuff that we do at the beginning and we'll do that in the second half of the show, uh because we want to talk to Scott- and uh you know basically learn about uh data and forgive me I looked at the wrong place on my screen here.

A

So data driven Fleet, Management, Scott.

C

Yeah, so we use a lot of data to help us decide. I think most people are probably familiar with. You know we have a fast Channel. We have a stable Channel. We use a lot of data to make decisions about when things go into, in particular the stable Channel, um so I think. That's that's a good title for or subject for this.

A

I like it, it was way better than the original title. I had.

B

Stuff and updates specifically.

C

B

A

Yeah, so so I guess first, can you tell us what your team does, um in particular, relating to the update, upgrade release process for openshift sure.

C

So I work closely with actually a few different teams, but in the context of this conversation, there's a team called the OTA or over-the-air updates team, and they were conceived shortly after openshift for GA and tasked with sort of monitoring, Fleet data and managing the the promotion process from the fast channel into the stable Channel.

C

um And so you know it's. That process has evolved over time and uh I've been involved with that since roughly before three or so, okay.

A

And and so when you say when you say, Fleet, Management you're, referring to deployed openshift clusters right and and kind of our customers and the things that they're doing as they go through. Those update upgrade processes, as well as all the QA QE engineering stuff. Inside of there right.

C

Yeah, so all of this actually falls after all, the normal QE process, um and so it's yeah. It's it's. How we manage the channels and um I think people are familiar with openshift 4 know that uh connected clusters talk to an API and that API gives them an overview of um what potential upgrades they could take, and that is a service. And since it's a service, we can amend that data over time, um and and so that's that's what that team is in charge of monitoring and managing got.

A

It, and is that Cincinnati, is that the Cincinnati.

C

Service Cincinnati is the protocol. um We we call it. The openshift update service is the sort of proctise name.

A

um And that I love our descriptive names, yeah.

C

That is, you know we. We run um an instance of that that clusters by default talk to, and then it's also um packaged up in an operator, so that customers that have like a large, disconnected Fleet can talk to a service that they host, and um you know largely it just mirrors the data that we provide.

D

C

But they could also do things like manage channels on their own um and you know, per their requirements and had in policy.

A

Yeah and just to give a visual behind that, so this is a 4.12.0 cluster that I deployed this morning. So when we come down here to Cluster settings- and we have this update, Channel update status thing- this is talking to an API on the other end and that API is what offers us. You know if I switch to we'll go with candidate 412.. So if I switch to something here, it offers apparently there's no candidate releases.

A

But it's what determines it looks at your cluster and and looks at what's available in the channel to be able to make recommendations and say you know, hey, you can go from 4.12.0 to 4.12.1 or 2 or 7 or 30 or whatever the versions happen to be, as well as the conditional updates right.

C

Yeah, so that's a new feature that we added with 410 clusters um based on uh amount of feedback, where um you know going back to what I said about we, we can amend the graph, that's presented to a cluster because it's a service and that that all hinges it around. Like you know, we go through all the normal process of QE, we test everything and then we send something out. And then you know some unique situation is encountered in the field and we find a critical flaw.

C

So we would prior to 410, we would go in and just remove that update Edge from the graph entirely.

D

C

Got a lot of negative feedback around you know it was it's very hard for us to manage that we had to manage that as sort of a big, huge binary thing. We either offer this update to everyone or we don't so if it was like. You know, a situation that only happened on a certain platform or a certain Cloud. We were holding back updates from other clouds, and um so we and then so. We just took a look at that and sort of the negative feedback of you know just a hypothetical we imagine.

C

We had a bug that only affected clusters on AWS. We in that prior to this, we would have to go in and like gcp clusters could not upgrade, even though they were unaffected by this bug. So in 410 we we said all right. Well, let's, let's fix this API and give it um what we call conditional updates, where we could say um all right: we've identified a risk and it affects clusters which match this criteria, and so we we query.

C

We actually present over the API, a prom ql, that's executed in cluster, to say all right. This is an AWS cluster, don't recommend this update um and it's critical also to understand the difference between um you know like in the past I said we blocked it like that. That edge would literally go away and when you wouldn't have any idea that it was there now in 410, we stopped recommending it and we, since the cluster, is matching the risk. The cluster is actually aware of these things still and it presents into the admin.

C

As you know, this is not a recommended update, but it remains supported because our policy is anytime. We publish an update Edge once it's published to supported channels and that's the fast and stable channels. It remains supported forever.

A

And sorry to interrupt, you I think you said something there that I really want to highlight as being important and that's that when a cluster queries the API, it's not actually sending any of that data to the API. Rather, the API returns back a promql query and, and the cluster queries itself to determine if it's eligible yeah that to me is really important.

C

The the only thing that we get back when you request the API, the graph API, is the channel that you're in your architecture and your current version, and then we just send back a list of potential updates and if any of those potential updates have a risk associated with them. Going back in this hypothetical AWS problem, we send that back and then the cluster itself evaluates that.

A

You say hypothetical AWS problem, I, know I know in in many people who watch the stream. There was uh several instances where vsphere was the the blocker for everyone, because there was some issues with I think it was the vmxnet 3. You know network adapter in the kernel and it was back in the four or eight time frame, Johnny, four, nine, something like that.

A

um So yeah, it's a very real problem and and something that I'm super happy to see. The conditional updates yeah uh so so Dwayne asks um how fast can an update, take effect and Dwayne. Please correct me if I'm wrong, but I I think you're asking how quickly does an update get published or how quickly does an update get applied to the cluster?

A

um So Scott I think it's an important question to ask, because you and I tend to answer these questions a lot in chat of how quickly updates move between channels.

C

Yeah um so I, you know we can answer both of those questions too. um So I guess maybe it's good to start from the the beginning of the process. You know we we go through a lot of engineering effort um and we produce a release and so that's promoted. Generally.

C

um You know once a week uh if we're considering like Z stream updates um and so that's promoted generally on Wednesday that goes to the candidate Channel.

C

um The point at which it's promoted is when we sort of hand it off for final QE regression testing and then that happens over um the span of like five days or so, um and that's once that has been completed. Qe doesn't find any regressions.

C

You know they're continuously regression testing, but we sort of give them a final build that they run one final round on um once that finishes, then you know the Errata go out and once the Errata goes out it's in the fast Channel um and then for in the context of Z stream updates. You know it's you're going from 410 0 to 410 1., that's a very small set of changes. Usually it's you know, 20 to 30 bug, fixes uh early on in the life cycle or product or if it's something like four eight.

C

Those Z streams will have like two fixes, because it's it's a very mature product and we're not changing much, um but either either way for Z stream updates those generally move from the fast channel into the stable Channel after about a week.

C

Now, if we consider like a new minor version update, um even so, we just shipped 412. uh I, looked at the diff stats on just the cube Brewery base alone, and that was over 2 million lines of code change.

C

um Minor version updates also occasionally bring in like Rel rebases. So you know that represents a Year's worth of Rel development. And so, if you look at a minor update, it's a huge change and so the amount of risk there is significantly more, and so that's why we actually do take typically around 60 days to promote a new minor version upgrade from the fast channel to the stable Channel.

C

B

C

Probably has like answers how fast Things become available in channels.

A

Yeah and I think it's important to point out a couple of things um and Johnny and Scott. Please offer or correct other things so, one like with a a y release, a minor release, so 410 411 412., even if it's not in the upgrade channel, it's still a fully GA and fully supported release.

A

um It's the the process of updating or upgrading from 411 dot, something to 412. something. That's where that's 60 days, 60-ish days, I know it. Changes um of testing and validation all comes in um and Hamza to answer your question. There's any difference between upgrade update and upgrade I'll say that it's mostly just a language thing.

A

um I personally use update to refer to a z, Stream, So 4.10.12 to 4.10.14 and upgrade to refer to minor version upgrades so 4.11 to 4.12.

C

Yeah um our documentation, we're centered everything around update and the the things that actually happen is exactly the same between a z stream and a minor version update.

C

um But obviously the content of the change is significantly different, um and then you know like not to dig too much into like how things happen like the CBO goes through and implies updates to a bunch of operators. Each of those operators can behave differently between, like a z stream and a minor version update so um but.

A

C

Fundamentals of how it works are all the same. It's.

A

Funny you say that because I I never really thought about it, but you're absolutely correct right, yeah, whether it's a 4. 10 to 4.11 or 4.10 dot, something to 4.10. something else. It's literally the exact same process. It's you know, CVO is replacing container images. You know which update The Operators, you know core OS gets patched the exact same way.

A

You know it applies a new RPM OS tree image, which comes from a container image and like it's just the scope of what's changed inside of those container images that are being you know, redeployed or applied is is different, as you said, you know, particularly when there's things like Rel rebases and kubernetes rebases, and all that.

C

um And if we want to get back to like how long does it actually take to apply an update, um what we've generally said is, regardless of uh the cluster size, the control plane update. We typically see that complete within somewhere between 60 to 120 minutes. So that's updating every single container, but it's not updating the OS for each node, the the OS update.

C

Obviously, when you deal with the workers that that's bounded by how quickly we can drain and relocate and reschedule pods- and it's also you know- you may have policy which prevents us from making forward progress because um you've set a pdb maximum available to zero or something like that, and so that that part of the upgrade process is potentially unbounded. And so when we talk about upgrades, that's that's generally.

C

What we say is it takes 120 up to 120 Minutes to apply the all the container updates and then the host updates can sort of trail on forever and having any sort of skew in that is generally not a problem. We, you know it happens during all upgrades, so we have to support it. So if you have, you know, 10 nodes which haven't updated. That's not a critical situation.

A

And are: are there you said that the the control plane kind of regardless of the rest of the cluster size takes a pretty consistent amount of time? Are there factors that can affect that and I'm thinking, in particular things like storage latency for etcd? You know if it's having trouble questing or something like that or if it's a super busy cluster right, there's a lot of things happening in FCD when the node comes back up and it has to catch up catch up.

A

You know super technical term um with any changes that happened while it was down. Does that affect it.

C

um I mean definitely if the cluster is under duress and not meeting, you know performance requirements, then that will slow things down. um Actually a lot of the time spent during the upgrade is just gracefully, terminating things and waiting for in-flight requests to complete, and so like you'll see, there's a very slow initial start. You know, STD rolls out. We wait very carefully to make sure that we update one member.

C

It comes back, it's healthy and you know we go on same thing happens with like the API server and the API server actually has like you know you, you shut one down and it will wait. I, don't know the exact timing, but you know up to 60 seconds or so for All In Flight processes to complete before it finally shuts itself down updates and moves on um so I think that's the largest sort of bounding Factor. There is how fast we can just gracefully do everything, and maybe that's one thing worth mentioning.

C

Is um we generally don't aside from you know, restarting your workers and relocating workload, um the control plane? We generally don't view as a disruptive upgrade or, and there we measure in our CI systems the total amount of disruption, and we generally see you know if we say that the upgrade takes up to two hours.

C

We generally see overall about 99 availability during that upgrade process, so that comes out to about a cumulative minute and a half um of disruption, while everything is rolling out and that that usually takes the form of like very short blips, where you know you, we do all the graceful draining and stuff like that, but certain clients may not. You know they have a connection, that's open and then suddenly that connection gets severed and uh you know you retry and it works immediately, but there's a little bit of disruption. There.

A

Yeah yeah the API server blipping, as it rolls pods and stuff yeah um Dwayne asked an interesting question here and it's one that um honestly I've never had a production or I haven't had a production. Openshift cluster openshift 4 cluster before. um But it's do you do you back up anything before an upgrade? um You know Andrew in my lab, I, just yellow it and you know hope for the best. But uh every.

B

A

Yeah from a customer perspective, it's it's a good question.

C

Yeah we, um starting with four nine uh at CD, actually takes a snapshot as soon as you initiate an upgrade and it blocks the upgrade process until that's done, um but backup and Recovery in a system that changes very quickly uh like any kubernetes cluster, like there's, constantly probes that are going on and pods are coming and going and volumes are getting attached and detached from nodes like backups, get very stale with the state of the cluster very quickly, and um so we, if maybe maybe it's worth talking about like rollback strategy and stuff like that, um if something goes wrong during an upgrade, our intent is that nothing should ever go wrong and compromise the availability of the cluster.

C

The upgrade may fail to make progress, but in no situation should we take a cluster down or or you know, because we do this graceful draining and we wait for things to come back. So while we we take backups as a precaution, um we don't generally recommend that anyone, if you run into a problem, immediately stop and do a restore or anything like that the backups are taken.

C

If, in the you know, I'm not aware of this ever happening, but if, for some reason during that upgrade the entire cluster just fails, and so it's a it's purely a disaster recovery thing and we don't recommend that um in in any situation, unless the cluster has completely failed, do you immediately start using that backup to do a restore.

A

That makes sense to me and I think for a couple of reasons. um One we've always said, and you know Johnny how many times have you heard me say here on the stream of at CD? Backups are not a full cluster recovery option they're really useful when you have one or maybe two of the SCD nodes that fail, and you need to bring them back up to a certain level so that they can resync. You know faster and get back operating that much quicker, um I.

A

Think if you were going to back up or protect something prior to an upgrade of openshift, it would be the applications, not not the openshift, because the openshift cluster in theory is disposable right, wink, wink we're all we're all doing.

D

A

um Yeah, so it to me it's you know the the let's say you know, like you said: Scott, you've never heard of it happening before and I trust you to be more in touch with that than anyone else. um But you know, let's say that the cluster did go down. You know in theory those pods, you know they would have Quest or they would have written data to a PVC.

A

That's gonna still be there on whatever the backing storage is when you re, you know, deploy a new cluster and redeploy the application, and you know container images and yaml definitions. You know if, if your devs are just going into the CLI, and you know yoloing an OC create command.

A

That could be an issue as opposed to using something more disciplined. um Even just a simple. You know shared file, repository of yaml files to help recover those apps yeah.

C

Yeah I think definitely like taking a more workload-centric view of backups is a good approach, because if you think about the LCD scenario- and um you took a snapshot now and someone goes and they cycle something and they had a volume attached and now it's no longer attached you're going to restore that and it's going to go or maybe that volume has been destroyed when it was detached.

C

Because that's typically what happens is you know you try to clean up data like it's going to go, try to reattach that volume and that volume may not exist in the cloud platform anymore, and so that you know, like I, said it's. It's a system that changes very rapidly and so backups are good to have. But they're not like.

C

You know your your first thing that you do if you run into problems during an upgrade yeah.

A

I'll say um and I don't want to speak for you Johnny, but I know you and I have a similar mentality of like.

C

A

An infrastructure perspective, you know virtualization background, we tend to think of protecting the infrastructure and you know being able to quickly recover that underlying infrastructure and openshift is not like that right. The the Clusters are by and large disposable, um and you know the goal is to get a new one deployed. So that way you can recover the workload on top of it rather than recovering the old infrastructure and oftentimes.

A

The workload would come along with that um yeah I'm, looking at looking at you, virtualization Solutions, no I'm, not talking about just VMware right rev is the same way. Openstack in many ways is the same way um so yeah.

A

um So I I wanna, I wanna, ask about um and now my mind, just completely went blank I I had a question and then I started to talk and then it left my head um I know: we've only got a couple minutes. Johnny I feel like I've been talking a lot. You got anything in the last uh four minutes.

B

No Scott, you covered it pretty well with the um like, when something's going wrong, because I I kind of have this feeling that there's a lot of people that are asking like okay. Well, what happens when something goes wrong? You know like specifically with etsud so say when the upgrade is happening. An entity decides like I'm done, you know it just blows up, and then uh it's like okay am I dead or do I need to do something, but it sounds like it. It shouldn't take down the whole cluster we should just be able to.

B

You know, maybe kill the upgrade pods and then uh let Etsy come back up and restore itself. Is that? Does that sound about right? Like? Have you ever seen an issue with STD being the culprit during an update, and then you can just roll back to the pre-upgrade.

C

um I think I definitely have and um again we whenever we Face problems, we just try to stop, and you know that the CVO never goes on Beyond updating it like SCD updates as one Atomic unit before we go on and update anything else. So in a scenario where we update one FTD member and um for some reason that doesn't come back, you still have two and you should still have a functional cluster and um generally, what we say is you know like.

C

Let's look at that member that didn't come back and try to figure out what's wrong, um you know you could potentially just say all right. Well, especially with the the newer features in 412 around control, plane machine sets.

C

You can actually go through and say well delete that Master and it would create a new one and hopefully that new one comes up and joins the cluster fine, um but yeah that that's generally, you know like everything in the system is built to remain available and when something doesn't work as expected, we just stop and we want to you know: that's when you you would call support and work through with them.

C

Or you know all this is exposed you you can look at your logs and maybe there's something obvious like maybe your disc filled up for some reason on that first node and you need to go and just delete some logs and then restart the host and then hopefully that CD comes back. Something like that.

D

A

Nice um so I, Scott I know you only have about two minutes left. Is there any anything else that you wanted to uh to communicate anything else you wanted to get out.

C

um I I know we said, like the typical minor version rollout is about 60 days.

C

um We try to keep it very data driven, and so you know all all these connected clusters are sending us Telemetry data, um so we pay very close attention to Telemetry data, as both clusters upgrade into the newer version, but also just as valuably um when new clusters are installed, because we actually find that the vast majority of issues are not like specific to upgrades. It's.

C

You know, there's just something wrong in the new version, and so it affects both installs and upgrades, um and so the Telemetry is a big input and but then we also scrape data from other sources. You know the entire support organization is dealing with customers in the field and they may encounter something. So there are other feedback loops and um Telemetry data.

C

Is it's machine driven, it's very reliable and it's very quick, so we notice problems there very quickly, but then there you know the the little the slower feedback loops of someone who creates a support case, and uh it may take a few days to get to us. um So that's sort of why things do take as long as they do. um We also need to.

C

We need to get a critical mass of new clusters to have all the data that we we need uh to sort of make those decisions, and so we have actually recently amended the documentation to say that we generally expect it to take between 45 and 90 days. But 60 days is actually the historic average ever since about four seven and it's been pretty reliably right around there. Yeah.

A

So I don't want to keep you any longer, um so Johnny and I will stay here. We're going to keep streaming for anyone in the audience, but uh Scott I know you have another thing to get to. So thank you so much for joining us today. We really appreciate you, uh you bailing us out and and agreeing to join.

C

No problem, thanks for having me.

D

Thank you so much.

C

Questions, let me know bye.

D

A

All right, yeah, so um yeah that Scott Scott's awesome, like I, said I've been interacting with Scott and our various uh chat and everything else internally for literally years answering questions for folks around updates and upgrades and he's he's always like on the ball, responding to things providing.

D

A

uh Engineering side of our engineering perspective, so I was so happy when he uh he said that he had time and it was able to join yeah.

B

Yeah he's also I assume all the time in our internal Channel I feel bad for a lot of those guys because it's like yes, the HTTP is required.

A

It comes with the territory right yeah, so it's a resetting a bit. um Let's go back to the beginning of the of the stream here. um So today is just a a day of Follies I, don't know! What's going on it's this line of storms, that's you know traversing the East Coast right, that's what it is we're in the upside down yeah, um because so you know the stream started. Our our intro video wouldn't play Johnny.

A

Couldn't change the theme of the of the stream that we're looking at today, like it's just been one of those days um so hello to our audience. Thank you for joining us. um You know. Apologies! We didn't open like this, but, like I said we wanted to get as much time with Scott as we possibly could um also um I feel like this is very selfish and random. uh Yes, I am standing today, uh I've been on meetings for the last.

A

uh Let's see it's 11 30, so four and a half hours I think um so it was one of those like I I made the mistake of not standing earlier in the day. So I was, you know it was just painful and I decided I'll stand for the stream. What's the worst, that happens right, um so it is uh good to stand up and stretch every once in a while, um but yeah so I hope it's not distracting to have this thing like covering and in the way and all that other stuff.

A

um Sometimes people get distracted by the background.

B

So far, it's been good for me and I'm I'm, usually distracted so yeah.

A

Well, it's funny. We were um so so Tech Exchange is next week right and one of the the demos or one of the things that we're showing is uh uh like a it mimics a dashboard of a car and through the windshields of the the mock car, is Google Street View, because it's mimicking like the car driving down the road, and we realize that we basically have to pause this.

A

Like you open the cars dashboards, you know the cars interface and like show maybe one or two updates and then pause it, because otherwise it's it's literally sickening because it's this constant motion and it's like you're, you know it. It jumps down the road because it's all that so I'm ever since you know we figured that out two three weeks ago, it's one of those like now I'm, all paranoid about. You know, motion being distracting and potentially literally sickening to folks. So.

B

It's like a 4D ride, yeah.

D

A

Have I don't remember where it was one of the theme parks when my my oldest was very little. We went to. It must have been a Disney one because it had Lilo and Stitch and you go in and you're sitting in this dark room and it's a 40th thing and it's meant to mimic um I think Stitches. The monster like him running around and like sniffing at you, so it like blows air on your neck.

A

This absolutely terrified my at the time like three-year-old son like he was not happy about this happening, and ever since then we avoid the 4D rides. Oh.

B

My God, we rode the uh the Harry Potter ride at Universal yeah, like the the whatever it is, the one in the castle and uh I was like I, don't I, don't get motion, sickness, dude, I, almost lost it. I mean like there's like 50 kids coming off this Hagrid's ride like right as I came out, I went in the club, it's like 30 degrees. Outside I, went in a cold, sweat I'm like taking off my hoodie I'm about it was gonna happen.

B

I was walking up to the trash can and this dude walks up to me and goes he goes you get. Do you get motion, sickness I'm, like I? Don't think so he's like you.

A

B

Now yeah any give me I can't remember what it's called the uh the pill. uh Anyways drama I mean yeah yeah, so he gave me some drum I mean and uh yeah that helped I was done.

A

Just gotta get you some of the butter beer. Oh that.

B

Stuff is so good, yes,.

A

Nothing like butterscotch flavored drink to that's.

D

A

Sell your stomach uh yeah. So this is uh sorry I'm gonna, completely change directions here on you, because otherwise, I'll completely Railroad and and you and I spend a lot of time just chit chatting with each other. We get to see each other in two weeks, yeah because you'll be in Dallas next week, yeah so yeah, um so yeah, hello to our audience. uh This is an office hours live stream, which means that we're here to answer questions uh so Scott talked about updates a few minutes ago. We're happy to continue answering questions about that.

A

If there's anything that Johnny and I can't uh answer, then we'll certainly reach back to Scott, as well as to the rest of the team to get those answers. But um don't let that stop you from asking about anything and everything you know 4.12 just launched, there's a lot of cool stuff. That's going on inside of there there's a lot of cool stuff that we're going to be talking about over the next set of streams um that we've got planned out so long as we can uh keep people from jumping around on us.

A

We did reconfirm Our Guest, for that was supposed to be for today for two weeks Ronnie three weeks, the 15th 15th, okay, yep, so yeah we'll be we'll be back after our travels and all that other stuff um hello, towel good to see you um I see Scott, keep in mind. The removal of recommendation keeps the update path available. Should the admin choose to accept the risk?

A

um Yeah? That's a that's a good point. um So.

A

With many of the updates and I think this is what Scott is saying as an admin, you can force an update like regardless of what the um what the tool says. You can always tell it to update to a specific version and get past that, but yeah conditional updates conditional upgrades, they're they're awesome, although I will say that it, uh the conditional updates, I think are have still broken the update or upgrade path. Interface.

A

Do you know if that's been fixed, Johnny I do not know.

D

I, don't have a I have a.

A

uh Handy to it, the upgrade path tool which we've shown before I just don't have a link for.

A

um Basically it's a it's a tool on access.red hat.com, where you can go in and you plug in your your current version and the version you want to go to, and it will say you know. Oh there's, you know here's how you get there. You go from this version to this version to this version or whatever that happens to be, but I think conditional upgrades have broken it, because if it's a conditional it shows up as not a valid path.

A

The admin has to go in and accept that condition rate except the risk associated with it. um So yeah, it's uh Chris, Ops I, don't know what chrissetops is I hope not.

B

A

D

A

I don't know if Christian's on I haven't I haven't seen him lately, um but yeah so uh anyways. um The upgrade tool is super handy, but uh conditional updates. The last time I looked. Admittedly it's been a few months. uh It wasn't uh wasn't working correctly.

A

D

A

Thank you. uh So let's talk about some top of Mind stuff um again we're flipping this episode around we're doing the first part second and the second part. First. um So let's talk about a couple of different things. um One I've seen this come up a number of times. I, don't know why recently, but several times over the last couple of weeks, I've seen it come up that vmotion is not supported, and quite simply, that isn't true.

A

I will admit, and I will say that we have had some confusing and conflicting, potentially conflicting information in the past and I will I will very readily admit that our wording could be improved, but in general and specifically compute vmotion is supported. So what do we mean by compute vmotion?

A

The docs here move to 412 and go to installing and scroll down here to vsphere and there's so many installation platforms now I remember when it was like five yeah.

B

It used to be really easy.

A

So if I go to this installing on a vsphere cluster and I scroll down, there should be I. Think it's in here and I usually find it vmotion.

A

So we have this using openshift container platform with vmotion openshift container platform generally supports compute only vmotion, so compute only here is referring to just a standard vmotion right. We're only changing the host that the virtual machine is executing on as opposed to storage view motion where it changes the data store that it's using at the same time.

A

So I realized that- and this is where I say it might be slightly confusing for some people, because a lot of people I think interpret compute only to mean only worker nodes, but the control plane is absolutely supported as well for doing a standard v-motion, a not a storage key motion. So why does this come up? Why? Why has this historically been an issue?

A

um So a lot of folks have some have maybe experienced some issues where in particular, a control play. Node will go unreachable or maybe NCD will um I won't say flake out, but where it'll have a couple of blips every time and when I've talked with our engineering teams and support teams like every time they have seen this, it has been a result of effectively a vmotion environment or a vsphere environment.

A

That is, shall we say sub-optimal uh so what I mean by that I mean things like there might be: Network injection congestion, there might be storage congestion right. There might be something that would cause that cut over time, CPU contention to be extended beyond what you would normally expect.

A

So VMware has a set of via motion best practices.

A

And this is what we you know tend to tend to point Folks at um I. Don't remember where the one of these is the right. One and I can never remember which one it is um yeah, that's a Dell. So it's probably this one but effectively. We just want to follow these recommendations inside of here, there's also a KCs article that I'll dig up here in a moment once I stop running my mouth. That points to like recommendations on how to ensure the emotions, so compute only vmotion is successful.

A

The vast majority of the times, um but I just want to reiterate I, want to stress vmotion, is supported with VM, with openshifts for all node types doesn't matter if it's control plane infrastructure right all of those things, it is supported um and it would be I, don't know it would be crazy to me if we didn't support vmotion, like that's one of the big things, I will say and Johnny you've heard me say this before that, and many folks who have watched The, Stream, Long Term have probably heard me say there is some consideration for pinning openshift compute nodes to certain vsphere hypervisors.

A

You know if you're, especially if you're over committing you want to manage capacity and stuff like that, uh the stream that we did around capacity management. We talked about this at length of you know. If my cluster, my VCR cluster, is over committed. um You know. Cpu memory, wise and I have multiple open shift. Compute nodes hosted on each one of my vsphere hypervisors.

A

You know what happens if a vsphere hypervisor fails. You know now it's going to be restarting those openshift compute nodes onto existing nodes that are already over committed that might already be experiencing contention and when those openshift nodes rejoin the cluster, it's going to immediately begin to reschedule workload to those. So it has the potential for basically exasperating. You know any contention issues any you know making the recovery longer all of that type of stuff. So in some conditions in in certain scenarios it may make sense to not be motion.

A

Those to you know, pin them to specific nodes, but that doesn't mean that it's not supported to be motion them to be clear. So I hope that helps.

A

um Let's see, oh so, the next one that I wanted to talk about here is uh vsphere vsphere version support. So it was pointed out to me this morning um that this is not entirely accurate, so vsphere 8 is still not supported from what I understand.

A

So just uh be aware that, as of this moment, um it is for for openshift 4.12 um 7.0.2 or later, but less than eight 4.11 I think it still works with um six seven I, don't know we're, saying 702 there as well. Okay.

A

So what I get for not reading the docs, it's studiously um yeah, so when I want earlier is now deprecated there we go.

A

um Let's see what else did I have I had another one. uh Oh minimum note sizing. It was just a random question that came up from one of the folks on my team. um They were looking for the minimum node sizing and we definitely don't make it easy to find because I, don't think I, don't think so. Here's minimum versions, but I, don't think anywhere. Do we ever say at least on this page?

A

Do we ever say like here is the size of the virtual machines that you need to use.

D

A

It's IPI and IPI: it just does it for you.

B

um Yeah I always look at like the bare metal or yeah the UPI.

A

Yeah minimum resource requirements for cluster installation, so yeah you have to look at either a UPI, install type or the non-integrated platform agnostic. um You know installing, on any platform and it'll tell you what those node requirements are. So it's it's not always obvious, especially if you're um you know, if you're doing like an IPI install, but then you want to manually add some nodes afterwards.

A

That type of stuff also note that if you're doing a compact cluster, you know so and a cluster where you're put it's only three nodes and you have both compute and control plane functions combined uh effectively. You add these two sets of things together, so the minimum node size for those three nodes in that cluster would be six CPUs and 24 gigabytes of memory.

A

um I mean I, hope my math is right there because I'm and they never do arithmetic in public. You only embarrass.

B

Yourself, public math is great.

A

Yeah um I just noticed that they added this iops column here too I. Don't think that used to be there. uh So that's good I guess I always feel like iops is the less meaningful um storage measure as latency is the more meaningful one, but you know glad to see that they're at least adding it uh there is a KB to calculate it. um I don't know what the context is there.

B

To calculate the size, oh.

A

Oh yeah, yeah yeah I'd have to uh unfortunately it is not listed and I have to dig it up, um which I'm not logged into the customer portal, so I can't uh dig around right now capacity, optimization stream. Thank you for posting that Johnny um 4.0 is AWS only uh yes, you are correct, but 4.0 was also never released.

A

um The first forward.x version was 4.1. You can see that in the doc in this up here uh a lot of people forget. We never had a 3.8 either.

B

I, remember that.

A

Yeah uh so yeah um so 4.0 was was only AWS and then 4.1 I guess we could go back and look here.

A

um 4.1, yeah, AWS, bare metal and vsphere were your options um and then I think it was four five was when vsphere IPI came along yeah.

D

A

Here we go sewing a cluster, so yeah trip down memory lane 4.5 um it's almost three years ago, yeah man.

B

A

And a half and a half years ago, time flies when you're having fun I.

B

Remember I was at Tech Exchange when 4.0 well, 4.1 was about to release you know, and so we're all like Coro was like what what's the operating. What what? What's all this craziness.

A

Yeah I um I was new to red hat at the time, uh because I think it released in early 2019 from what I remember.

A

um So it was like six months after I had joined red hat and I spent six months like learning, openshift, four and operators in the install process, and it's like I get I got super frustrated with um all of these things and like because Opera like operating. Why do I need an operator? Why do I have to go here to change this thing when I can just go in and change it in the in you know the config file and yeah. You know having to make that mental shift.

A

uh Ryan um really know your open shift history. uh Thank you. um I I, it's more of a I've just been around for a minute, um so yeah it uh and I also focused uh so installer stuff.

A

I is a little bit cheating for me because um for a long time uh as a tech marketing person, my my one of my SME topics was the installer I used to focus on insulation, core OS uh etcd, like all networking kind of all of these low layer, things um and then I've always relied on other folks on The Wider, Tech marketing team for expertise around like application, Level stuff um get UPS like Johnny here my you're, my get Ops go to now Christian's dead to me so.

D

A

Hope he's listening too yeah, or maybe this is going to YouTube right. Can we like clip that on YouTube and and send it to him it's from him, I.

B

Need to figure this short for this episode.

A

Now I need to watch my slack counter down here in the corner.

D

And see if he starts messaging me.

A

On slack awesome.

A

Yeah so I see you dropped a couple of things in here as well. um Please.

B

uh Yeah, let me bring it up really.

A

B

Yeah, so um the there was really two big announcements this week, I think they're kind of big, but like the the first one was red hat: open data, Red Hat, openshift data science or roads, um they're on-prem support just went GA, so Rhodes was a cloud-based service um and if you don't know what Rhodes is it's essentially uh the data science uh capability as a cloud service?

B

So you get things like Jupiter notebooks, um there's some other public notebooks that they put out there, but all the things you need for AIML, workloads and stuff, like that. It's built around that and uh they've moved that from not only being a cloud offering, but now to an on-prem offering another big key thing coming out of. That is that right now, airgapped isn't supported, but at in this next release in the next.

B

Like you know a few weeks, it says end of July or excuse me end of January, beginning of February um they're, going to release air gap support. So if you have uh Jupiter notebooks or you're doing data science capable tooling on your clusters in an air gapped environment. That stuff is all uh it's on the way um and they're they're going to follow some of the same releases as the cloud services.

B

So you know if they, if they release an update for the data science cloud service, then the on-prem will have that same update and that's generally about every three weeks or so um and then the other one was openshift pipelines. 1.9 uh our hope9 invention, techton uh 1.9, is out so there's a lot of GitHub and um uh other like sem providers, The Source control manager providers, so webhooks for uh bitbucket, uh gitlab and GitHub are now all supported, which is pretty cool.

B

uh There's just a bunch of cool stuff coming out with openshift pipelines, uh there's some PR status report, and then uh you can do events. If this one was a cool one, that I thought um events and filtering. So you can have a pipeline per event. So as like, maybe you have an openshift serverless trigger, then you can have a pipeline or whatever I, don't know how it works. I'm, not that I'm, not that uh application-centric, but um but just a bunch of cool things that are coming out with the pipelines.

B

This code feature to just enhance it and make it better um and then the last one is there's a Blog for what's new in ACS and it just kind of going over some of the things that are coming out.

B

The one big thing that I thought was super awesome that I I can't wait to actually um Kick the tires with is the um the automated Network policy generator so before you deploy something it'll generate a network policy for you and uh it'll be available in the ACs console, so I think that that's just going to be amazing. So looking forward to that and then the other ones are just Ubi 9 support and uh improve vulnerability management.

D

Ubi 9 yeah, yeah, yeah, crazy.

A

I, don't I, don't even remember when does openshift when does core OS switch over to rail 9 base, I, don't remember, I.

B

Have no idea yeah soon.

D

Though I I see.

B

Him like talking about it so so.

A

Yeah that'll be happening sometime, that's kind of cool um I also think ACS they're also moving away from the Rocks DB and over towards postgres. Now um so I I think I mentioned that before of I, never knew it used a quote: unquote: proprietary database um I, don't know if it was literally proprietary or just like a modification to some some random.

D

A

Yeah but um I I never knew that so that's kind of cool yeah. um What else was I gonna say there was something else: uh oh uh Dwayne wrote. A so wrote is something different. um So Rhoda is Red. Hat openshift database database.

B

A

Yeah, um so that is, that is more or less. Let me think of a good way to describe this. It is a service that you can go to so basically, your your databases would put connection information into Rhoda and your applications can then look to Rota to get access to that. So it acts like a broker for them to get access into resources.

A

Rhodes, Red, Hat, openshift data science is a uh well historically until now it's been a managed platform for doing data science type work, as Johnny said you can go in you can you know, click the thing and say: I want a new Jupiter, Hub Playbook and you know I need x, amount of CPU memory and GPU and you click the button and you're dropped into a place where you can immediately begin working.

A

um You know you don't have to it's a simplified interface, it's a streamlined way of getting data scientists, access to resources and with it being on-prem.

A

um That to me, is um it's interesting right because it it will hopefully allow those data scientists to focus on. You know data science and not having to learn kubernetes or not having to learn openshift vernacular. They just go in and click some buttons and their stuff works. Yeah.

B

Exactly and having the image streams and all that stuff with those notebooks available or what the way for you to like, hey: here's, where my notebooks are. If I already have some and it'll ingest them in the image and then the Rota will actually pick that up and then your stuff will be available within the toolkit, which makes it yeah it's just really streamlined and really nice.

A

Yeah yeah um for Red Hat folks, who are watching so there is a open, hybrid, Cloud demo available uh that includes Red Hats uh uh uh openshift data science inside of there. So you can go and get Hands-On. You can look at it and you can do all the things with it, including run through a couple of different queries and update machine learning models and all that other stuff.

A

It's really cool I, think there's other there's several other dedicated ones as well, but I just I favor, the open, hybrid Cloud one, because it's sort of an end-to-end thing, which is neat I'm, also slightly biased I, may have participated in it.

B

A

Some skin in the game. uh You know, you know uh it's uh I'd say it's Shameless self-promotion, but um well it is Shameless. It's not just self-promotion, because I only had a very small portion of it. So it's it's uh promoting other people's work too. There you go um yeah.

A

So let's talk about a couple of things that are coming up next Johnny, um so one we have, and let me bring up my page here so one uh as we sort of alluded to um earlier today as- and we mentioned it last week, uh Johnny and I will be both traveling uh for the next two weeks. So the the live stream ask an openshift. Admin will not be streaming until again until February 15th, um so we will miss you all um again.

A

I I said this like before um it feels so strange to like basically over six weeks, we'll we'll have done two streams. um We're seven weeks. We'll have done two streams which feels super weird to me, um but yeah it's! It is what it is right. We we gotta gotta, do what we're you know paid to do.

D

A

um I have a tiny bit of a Spam issue. Did I miss a Spam thing.

B

Oh yeah I I got it I got it as soon as I could I I saw it in restream once, but it was already in twitch like four times yeah.

A

Okay, yeah, we'll we'll um we'll take care of those.

D

Do I have oh there we go, I can delete them out of. uh Maybe I can delete them out of here.

A

No, it tells me unable to delete message all right, we'll sort that out. um Thank you for letting us know, Rick appreciate that um we try to keep it, uh keep it under control, but sometimes you know Bots. It is what it is uh so two weeks from now, February 15th is when we'll be back um Johnny. We we think, or that should be uh logging right. I saw somebody mention. Openshift logging was updated.

B

um Yep, that's our hope. Nine calling out that uh 5.6 just got released as well and uh good call. Thank you for that, because I miss it um and we have the product manager from openership logging on the 15th she's going to come on and tell us all things and I think she's going to do some demos and stuff like that. So it's been a long time coming we're looking forward to it.

A

Yeah yeah there's been um lots of questions lots of interest on logging, particularly as we make the switch over to Vector, which is now fully GA I, think maybe even the defaults I think it was GA in fortnite or logging, five five and now it's the default with logging, five, six, five six, maybe I'm, don't.

D

A

That yeah, don't quote me on that I'm I'm, remembering things that uh went passed by you know zooming by an email, um but after that I don't think we have any major interruptions like not until Summit, which is in May, so yeah we'll have like two months: two solid months. uh You know, like eight streams, where uh you know we'll just be talking about all things, um 4.12 and kind of anything else that comes up inside of there I'm. Looking at our list of things that we identified.

A

So uh we know we want to talk about the disconnected agent based installer. um Somebody sent us an email about that. Thank you for the request, um or maybe it was a message. I, don't remember uh the ebpf network observability.

A

um This is one that I know very little about, but I'm super excited to uh to learn more because it looks really really cool.

A

um You know, being able to see what's happening inside of the network and traffic flows and how things are going between pods and all that other stuff. That's really awesome, so I'm, looking forward to that um no FIP support for vector and five six. Oh that's disappointing!

A

um The security profile operator. uh Topology aware lifecycle manager, um that was another one I I did not know that existed until the uh the the what's new session and I saw that and I was like super impressed. I even sent a message over to the team and slack and was like this is awesome. Where have you all been like, so we definitely want to get them on. We definitely want to talk about um the topology aware, lifecycle manager stuff. All of these things there is the little blurbs in the uh what's new.

A

um So if you go back and watch the what's new stream from two weeks ago now, um you'll you'll see all of these things just in little blips we're going to Deep dive into them as we get to these streams uh logging. Let's see what service mesh see if we can get ortwin back um to talk about what's going on with service mesh and then uh I'm gonna hit up Mr John Spinks again and see.

A

If we can talk about some new stuff, that's going on with uh openshift uh Red Hat insights for openshift, um because there's lots of cool stuff going on there too yeah.

A

Ebpf is basically Blackmagic, um I I, you know, I've been a Linux admin for, like literally my whole career. You know 20 plus years and there's some things that happen at the kernel level like ebpf, that it it is a black magic, yeah I agree with you Rick, so I'm I'm kind of interested to get a little more knowledge and get you know, make maybe get my hands a little dirty and figure out uh what that looks like and what we can. You know learn about it.

B

A

Either that or I'll just become a regular consumer and just oh pretty charts: let's go you know: I I, like pretty charts, I, don't care how they got there, I'm. Okay, with that too yeah.

D

Pretty exactly all.

A

Right um so yeah I don't have uh uh I, don't have anything else. Johnny I think we kind of rolled through everything that we had um I'm pausing here to see. If anything comes to mind and also to see, if you had anything, um no.

B

I I think I think once we get back, it's just it'll be a whirlwind again. You know, but um looking forward to it, I think that there's gonna be a lot of good things. uh There's so much coming out that we uh that we have to talk about and having having everybody come on and show us the uh show us the robes and answer questions I think is going to be a lot of fun so um but yeah for.

D

B

I'm in I'm, in like trying to get into presenter mode, you know so we'll see.

D

A

I'm with you I'm uh my morning today was taken up by by rehearsals, uh because we're doing you know the keynote, so you know had to go through and do the rehearsals and check our environments and validate everything and all that other stuff. And it's uh it's been a few years since I did a keynote.

A

um You know because doing it virtually is just not not the same, and so it's a it's it's fun and it's challenging, um but it can be a little exhausting um so I'm I'm, happy I'm I'm happy to be involved uh because we get to show off really cool stuff and that's why we're here, um yeah I, agree: I.

D

A

And and I think uh you know we I I'm I need to check I need to find out if we can uh basically do all that stuff here on the stream, um because it's all stuff that's possible today and that would be really cool. That would be.

B

Cool, oh man, yeah. That would be awesome, especially like some of the talks that didn't get accepted. You know to take some of those and maybe do some of those on the stream as well. I think that'd be awesome.

A

Yeah yeah or hope nine making a plug for ACS there. Thank you, um I I promise, I, don't forget about them on purpose.

A

All right uh so to our audience. Thank you so much for joining us for this slightly different stream. Today, um again doing the second thing first and the first thing. Second, uh we hope that this has been useful. uh If you have any questions um right now, while you're listening to us, please go ahead and post those in chat we'll do our best to answer them before we end the stream. If you're not listening to us, live by all means, send us questions.

A

uh Johnny and I, both love to get messages, emails Etc from anyone and everyone. So you can reach me at andrew.sullivan, redhat.com via email, that's probably going to be the best way. um I hope not I. Think I have an email from you that I need to reply to I admit that I have been way way like way, far more than usual behind on email, so yeah I will prioritize that and get a response to you, um but yeah so reach out via email uh on all of the various socials at practical Andrew.

A

I admit that I'm avoiding Twitter right now, um because it's very noisy, like I, haven't figured out how to get back. My old timeline of just showing me the people that I subscribe to in a chronological order, um but that's a user error, I assume but.

D

Yeah otherwise,.

A

uh Let's see what uh uh Reddit and all of the other places, um you can also join us on slack, so the kubernetes slack has an openshift users channel that you can join and Johnny and I, along with like 4 000 of our closest friends, are all in there. It's a great place to get help and then, of course, you can reach out to Johnny as well Joe and NY, red hat.com or Jrock tx1 on all the various socials, and uh thank you for taking care of social speaking, speaking of which today Johnny well.

B

Thank you for everything else.

A

All right, Joe, um so yeah. Thank you, everybody. We really appreciate it. um We will see you in three weeks on February the 15th uh where we'll be talking about logging and with that I will uh leave any last words to you. Johnny.

B

Yeah good City again I'll see in a couple weeks um just for everybody. That's watching. uh We will not have the outro because it doesn't work right now, so uh we're just gonna end abruptly, so uh not good stream, thanks Scott, if you're, if you're listening. Thank you so much for really the last minute uh accepting and coming on and helping us out like this was really awesome and uh and really informative. So uh yeah, that's it for me all right! Bye, y'all, later.