Red Hat OpenShift The Cloud Multiplier | Red Hat Livestreaming, 11 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Cloud Multiplier: Ep. 9 | Confident OpenShift Upgrades with the Update Graph

Description

Kubernetes is the resilient platform for your containerized applications, touting concepts like zero-downtime rolling deployment updates and metrics-driven application rollouts, but what happens when you need to update the platform? OpenShift, powered by the Update Graph, delivers a data-driven approach to platform updates. The update graph transparently provides insights and update guidance across architectures and deployment topologies - from on-premises to edge. This week, Lalatendu Mohanty and Pratik Mahajan from the OTA team join us to explore the inner-workings of the update graph. Join us to learn how OpenShift leverages fleet-wide analytics and CI to make data-driven changes to the update graph to provide every OpenShift cluster with a seamless, confident upgrade experience.

A

A

A

A

A

A

Hello and welcome to the cloud multiplier your show for all things distributed: Cloud I'm, your host, Gurney Buchanan and, as always, I'm joined by my co-host Jody Banerjee. uh Welcome back uh and I have the pleasure of welcoming pratik and law attendu here today from the OTA team. So that's over the air up. The updates. These guys are on one of the coolest teams in the openshift engineering organization. I'm, so excited to have them here today. As always, we'll go around the room um and talk a little bit about what we have going on.

A

I have been very fun in devops land and outage land uh debugging a nice incident this morning Jody tells me that he's been doing some exploring with metrics right metrics forwarding to Kafka.

B

That's right, I've been playing with you, know, metrics forward into Kafka and then putting a spark on it and seeing what we can do, and one of the pleasant surprises was I mean this is this is new to me, but it might not be known to the community uh spark now with the integration of the kubernetes, makes it very easy to get actually a fully functioning loaded spark running on Cube, like this I.

C

B

C

It when you had to set up.

B

A

Operator based.

B

Oh wow, it's like this. You know you can submit us, you can do a spark submit job directly to the food master. It just takes care of it. It is awesome actually, and if you recollect, you know Ernie, if you ever try to sort of spark using a yarn custom manager or whatever it was much more harder. It's all kubernative! So that's! That's! uh It's wonderful and I've been having fun with that and the the other thing you know you've.

B

Given me, a soapbox, I've been I, have read a wonderful book called the two percent way. It's uh it's extremely inspiring it's written by uh a guy who is uh now a neurosurgeon. He is still come. Maybe he has just become a doctor resident at Harvard, a road scholar.

B

He was a full-time great. uh This thing at FSU, a safety he made it into uh NFL, but he never really got selected for the play. You have to read the book to understand what what the reason for that was. Can you imagine a NFL player, a roller, a neurosurgeon and if you read his book through the obstacles that he has gone through in life and how he surmounted them I'm like stop complaining get here.

A

I think I I think I found it on on our favorite place. Amazon. uh You know the only place to find books.

B

Only place to find books, but that was you know so in between spark and Myron and the regular. You know regular stuff, but it's been, it's been exciting and.

A

B

Weather is changing it's getting even more exciting.

A

I, don't know how you find all the time for it well again, guess we'll follow that up uh we'll go around the room. uh Do you want to tell us a little bit about yourself, introduce yourself.

C

Yes, hi everyone, um I am the mohanty I work for the openshift OTA team over there update team uh I. Am the team lead for this team? For more than two years now uh we have been working on very interesting things.

C

I'm I'm really excited about the thing we work on with respect to openshot upgrades and I think we are doing some things which kind of leading in the industry right now, which obviously some of it will talk about today and um so yeah I'm I'm, based out of Raleigh and uh right now is, as of now I think I'm working on figuring out.

C

um You know when we can put our upgrade edges from photo 10 to 4.11 in forward 11, stable Channel, trying to analyze the data and trying to figure out when we can do that. uh So yeah, that's about me.

A

Amazing, as a stable, Channel user, I'm hotly awaiting the announcement, so we'll we'll be doing exactly the same, we're starting planning uh fatigue. Do you want to introduce yourself as well we'll go around the room? Oh.

D

Yeah uh I'm pratik Mahajan I am part of the openshift um over the air updates team as well. uh I, primarily work on the Cincinnati protocol and on the openshift update service operator, um so managing the production uh for openshift update service. What you called ursus internally!

D

um That's what I do I work work on developing the back end for update service and just uh like maintaining its uh availability for our customers so that they don't like uh save any outages so yeah. That's what I do.

A

That's amazing. Well, so, on the show past couple times, I've realized Jody. We've I've accidentally made a very good series of shows in order. We've just happened upon folks in the right order. So last time, I I believe the last time we're here we talked about insights, we missed a show from PTO.

A

Last time we talked about insights, which is really about making data driven recommendations to your Fleet um and- and we talked before that about various ways- to make a fleet and manage a fleet that is, quite frankly far too large to treat every single cluster, with the sort of focus that you treat one very large cluster. Those are still very important.

A

One very large cluster is incredibly important, but what we're talking about today is a piece of technology that gives you confidence in upgrading your openshift presence, whether it's one incredibly large cluster running on premise or it's something running in the edge or it's something running on a public cloud or it's a thousand clusters running in a thousand different places and and making data-driven decisions. So I'm incredibly excited to hear about this piece of technology.

A

Do you guys want to kick things off I know you have a bunch of information and cool repos to share with us yep.

C

To share the screen, yeah.

D

I'll share my screen and then we can take it away from there right. Can you see my screen I just there we go okay. Let's start if.

C

So I I'll start the slides I'll talk about.

D

C

Slides so hi everyone, so we're talking about openshift upgrades today and how openshift upgrades you just upgrade graph. um You know the whole thing. um Obviously you know a lot of Concepts. We kind of break broke it down into multiple slides. So let's go to the first slide.

D

Just a minute, oh.

C

Okay, so um the first thing is: why, right, why openstitute of grade and uh like how it is different from kubernetes, because um I personally think openshift upgrade is actually a way it's a lot of valuation. On top of uh you know, kubernetes like um and uh I mean anybody who is talking about how openshift is different than kubernetes. The upgrade discussion will definitely will come. It's definitely one of the important components of how openshift adds value versus a vanilla kubernetes.

C

um So if you have used a vanilla, kubernetes or if you have not the.

A

C

In kubernetes, when you have a multiple node cluster, let's assume you have a cluster upload, maybe 10 or even 20 or 100 nodes. So many of our customers have 100 node clusters, and um so, when you upgrade equivalent is cluster, you have to upgrade the nodes, because you have to upgrade the kernel and every all the packets in the distribution like Linux distribution, whatever you're using, and then you update the kubernetes um runtime to do that.

C

Obviously, you can manually go and change 100 update, 100 nodes right, you might use ansible or some tool to basically do that. Still it's a lot of pain. It's very hard thing to do.

C

um It's gonna, take a lot of your time um like you have to babysit the whole thing, but in open shaped, the upgrade is actually a combination of upgrading the kubernetes open shaped, which is basically equivalent distribution and also upgrading the nodes. So it's a single key upgrade. You click the upgrade button and the whole update starts and when it's done the nodes and the openshift is actually upgraded and there will not be any workload disruption during the upgrades. That's why we call it over there update availability takes precedence over everything else.

C

Why I am saying this you'll? You know I'll talk about this more in later slides, but we give availability the highest priority. Then everything else is a secondarity. So let's go to the next slide.

C

um So before we go into the upgrade a little bit of context around openshoot architecture, because that gives us more context around um how we're doing the things openshift is a operated distribution of Linux distribution. uh Sorry, a kubernetes distribution. uh We take kubernetes, and on top of that, whatever uh valuation we do right, we run them as operators, so openshift container platform. We call it RCP version. 4 is nothing but kubernetes and there are a bunch of operators running on top of it. So it's basically kubernetes.

C

We extend kubernetes uh for our purpose and we do it using custom resource definitions and operators. Obviously you know this information needs to store in the API server, so we have we're running another API server, which is called openshift API server. So we run two API servers. One is Cube API server, one is openshift API server, so whatever vanilla Concepts uh coming from kubernetes are actually stored in qbi server and the other thing extension or the API.

C

We have extended lives in openshift, API server, and we have lot of operators and cluster version operator, which we also maintain, and you know um often should allow OTA team actually maintains our team maintenance is responsible for starting the upgrade, so cluster version operator is responsible for many things during the upgrade it's kind of the um you know the orchestrator kind of situation where he, the customizing operator, kind of, starts the process and also looks into the process making sure everything is going in the way. If not, it communicates the account user about.

C

What's going wrong, uh we'll talk about more in the next slides.

C

Operator in principle have lot of status conditions where how you know through status condition, they show a lot of information to the users. What's going on during the runtime and I put here three status condition, which I thought was important from the upgrade point of view, one is availability which is called available. It can be true or false if it is false. That means the operator is not available. uh It's true. Obviously it's available, and there is something called degraded degraded is where the operator is not its hundred percent um Optimum State.

C

Something is going wrong for a period of time and that's how operator communicates it's kind of degraded, so degraded me. If your operator, if an operator is degraded true, it does not mean that it's not available. It's still available. So I'll give you an example. So, for example, let's say we have an operator which needs four replicas to be run always and some for some.

C

For some reason, only three replicas are running and it has been in the state for let's say whatever the service level SLO defined for that operator, which is maybe five minutes of a period of five minutes till running on three replicas right, then the operator can go that yeah. I am actually integrated now because I'm not supposed to before replicas now I'm three replicas.

C

So that's uh that's a degraded uh condition, and um that would mean the operator is still available, but maybe the quality of service is still less and once this goes to degraded condition, then you know, because it's operator model kubernetes, always trying to reconcile to the desired state. It will keep on trying to reach to four replica and once it reaches the degraded conditional go away and things are again good, um and so the grid is very important from that point of view.

C

Many times many times, I've seen customers, even you just complaining about my operators, degraded and they get. You know panic, because it's like yeah something's wrong with my cluster many times. It's actually not an big issue and obviously availability is not impacted. So don't worry about. You know the workload disruption will not happen.

C

Then there's the another status condition is called upgradable. It can be true or false uh when you are doing upgrade between one minor version to another minor version. That's why it comes into picture. um Let's go to the next slide.

C

Because upgradable is a very important uh condition. I wanted to talk about this in one slide, um so it's only impacts. Miner updates, uh many times when you are trying to upgrade from one version to another version um and if some condition doesn't ma made, you know it doesn't some condition doesn't match.

C

Then it actually go to upgradable false, um and that is a that's a that's a good thing, because you're trying to protect the cluster uh so that you know you don't get into anything else like availability is a hard priority and that's several conditions when it's got upgradable. False, for example, if there's API, um if the operators operating to another version, it's fine that there is something um wrong with the next version or it the code can do anything.

C

So it can figure out that yeah I'm, not in a state where I can upgrade to the next version. It will set up and hold false. That's a good thing and you upgrade uh will not happen, will first fix the issue or the issue might get fixed itself and then we'll move to the next version. So it's, but only thing to keep in mind is that it only impacts. The minor update, which is from 4.6 to 4.7 or 4.8 to 4.9 4.924.10, not in between the G stream, operates like plot between 4.9.1 to 4.9.2.

C

Okay, let's go to the next slide so many times you know when you do upgrade, things might not go uh as planned.

C

um Obviously the success is really high, but sometimes it might go how you you know it might not go how you plan things and how to react to that situation. So in general, during a period operators should not go to degraded State. That's uh you know. During our testing and other things, we make sure that we're also trying to um like restrict. We want to be more strict about it, but no operators should be going to degraded State during the upgrade.

C

However, if things go is unplanned, maybe some node is not available and you know the network stack happen. Network switch happens, there's something wrong with the network switch or anything with a network, and uh um you know operator obviously continuously reconciling its figure out such as something around right, so it can go to degraded state and once you go to degraded state will stop.

C

The upgrade will not progress with upgrade and wait for the degraded condition to go away, and- and if you see the cluster version status conditions like OC get cluster version, you will actually see all the information there. Also it's get, it gets pushed to the web console so web consolows. Us you the same thing and uh because, like I said in the previous slides, openshift V4 is actually composition of many operators, and if you do OC gate cluster operator, you can see all these operators showing their state.

C

You can see their version and you can see their available true progressing faults. Maybe some some of it is getting updated. Right is Cube, API server is getting updated, so it's basically true and if they are degraded or not so as long as you're available is true, always like I mean, if you see our variable is true, your workload are not getting disrupted and there is a sequence through which each operator gets updated. It's not all operators doesn't go at same time, because the cube API server is just updated.

C

First, it's D needs to get upgraded and then gradually you know everything get upgrade and at the end, machine config operator actually upgrades the nodes, because that's important right. It should train the workloads from it. Reboot the nodes and again the workload comes back, and so this OC get cluster operator actually is the right command for you to see. If what is the state of your openshift components, are, they are all available.

A

Is the component upgrade order, also part of the cluster version operator? Is that also something that gets controlled there? You know this operator upgrades first, then this one, then this one.

C

Yeah we have an cluster region operator, we have something called run level and whenever we add operator as a core operator into openshift, you have to define the Run level, and that is where our cluster version operator applies manifest. So it first try to update the cube API server right, yep and the way it works is opens uh the cluster Vision operator downloads, the release, payload and then catch the Manifest out of it and then try it. It actually applies the new manifest for cube API server. Now, once the IPS server manifest is applied.

C

The container the operator which actually manages this in cluster resources from coming from the Manifest, but actually no yeah I- need to move to the new version, so they download the new image and then upgrade themselves and cluster version operator, basically monitors them like okay. If there's anything wrong, they can push the information to Cluster version, monitor customers in operator and it's the status get collected there.

B

uh And you use the word, monitor uh probably uh slim of your talk, but yeah was do we um do we actually monitor how much time it takes to up to do upgrades I know it cannot be generalized at all five node cluster versus 100 node cluster upgrading from four eight, two, four, nine and four nine to four ten or four nine, two, four, nine one totally different piece, but do we have any? Do we collect any Telemetry data for that? Actually, yes,.

C

We actually collect Elementary data and uh we try to see every for every release. We try to see our average upgrade. Time has increased or decreased and to compare it we need to be like you know, it's we need. Let's compare. Cluster Which is less than 20 clusters or maybe 10 number of nodes. Obviously it depends on how many nodes you have, uh or you can also compare, only the control plane like how much time the control panel is taking to upgrade I.

A

Would definitely.

C

Do this kind of.

A

Thing so when I say monitor.

C

It's not exactly like the Prometheus monitor, but it's more like you're reconciling so it's you know it's operator even driven thing. So it's trying to see the some informations are the English resources, some information, it's not! It just sends an event and the whole reconciling starts again. So at that point thanks for catching that, uh let's move to the next slide, please so open shift.

C

You know we talked about how upgrade happens right, but how does openshift know which version to upgrade so that information comes from a API service which is called open sheet, update service and red hat actually runs a public instance of openshift operate service and each openshift cluster. If they're connected to Internet the cluster version operator from that clusters, pulls the open should update service about you know this is my version.

C

um What version I can upgrade to and that pool happens every two to three minutes.

C

um You know there is a yeah time um time defined there and we keep on asking API server and once we get the information like the cluster Vision operator gets the information and basically then it populates the available updates in your cluster.

C

Okay. Let's go to the next slide.

B

So what you're saying here essentially, is that if I'm a customer running openshift I have a cluster which is connected to the internet, then my operator is pulling a service from where information gets downloaded, which is very prescriptive to my environment. Right.

C

That is right and and how we do that we're going to talk about in the next slides, yeah, so pratik.

D

Yeah I think we'll talk about the next slide. Yeah I'll take away I'll, take it from here. uh So as Lala told you right, we know how the cluster upgrades, but what like? How do you tell the cluster that there is an update available? So what CBO does is it will pull the openshift update service, which we call Asus, and we have a public instance running at api.openshift.com and that's the default. uh It goes.

D

It calls uh the update service in with its parameters, so, like I am um AMD 64 cluster I am on this channel, uh so I want like a uh I'm, unstable, 4.11. Channel now I need to know what is my next version. That was the next version that I can upgrade to so we use a directed a cyclic graph dag in order to store that whole data. So, as you can see, there is like a small graph on your screen, so it says like uh there's a version.

D

So let's consider if you are on version 1.0.0, so you have now two edges. uh The first Edge, which is the upgrade Edge, is what we call uh so the first Edge is 1.1.0. So from 1.0.0. You can either upgrade to 1.1.0 or 1.3.0 and, as you can see, there is no upgrade Edge from 1.1.0 to 1.3.0. So if, let's say, if you are on 1.1.0 and if you want to go on 1.3.0, you will have to go through 1.2.0. So that's what the update service basically does.

D

So the cluster will ask and then the update service will give it a path like this is the path you need to follow. So that's what we do and then we have two dependencies. The first one is the release images which stores all the metadata about the release, and second one is the Cincinnati graph data with Cincinnati graph data. What we do is we can manage the graph. uh So let's say if we wanna promote a version from Fast to stable Channel. We do that in Cincinnati graph data.

D

I will talk about that uh later in next slides. So let's go to the next slide and uh does anyone have any questions uh till now.

A

I think you're about to probably answer my question, but I'm always curious about how you end up as someone who chips a thing that only has one Z stream. You know you go from 2.4.1 the 2.4.2 to 2.4.3 I've always been interested about those skip versions. Those where I can go straight from you know, openshift 4.10.30 to 4.10.34 and I can make that big jump and I'm always curious about how you find those edges.

D

A

So I'm sure you sure that's probably like the next slide. Yeah.

B

And there's a question on the public Channel by the way.

A

Awesome yeah there is a. There is a question.

D

C

The question is Red Hat Public instance provide metata, include virgin and URL for or just the Virgin compatible with my open ship, so we actually have in the later Slide the exact crab snippet. How does it look like, and the whole information basically comes to the cluster, so yeah we'll get to that yeah yeah in case it's not answered.

D

Yeah but like before that, we'll just look into the working of how the Cincinnati protocol exactly works. So what Cincinnati protocol is is that it is. uh It just describes a particular method for representing a transition from One release to, and just like, uh as I explained with that example before so it has two components. The first one is the graph Builder, so what graph Builder does is.

D

Basically, it builds a whole, a big graph that has all the channels all the versions, all the architectures, and to do that, we get the primary metadata from your container registry for Upstream we use query.io and, like it also gets the prime is it gets the secondary metadata from github.com. So we have like a Cincinnati graph data repository, so we pull it from there or you can have this secondary metadata in in your inner container format as well.

D

We'll talk about that when we talk about the openshift update service operator, but these are the two places from where we get the metadata. We build the graph and what policy engine the second component, which is policy engine. What it does is. It will trim the bit graph for your cluster like, depending on what request you send. So, let's say if it's AMD, 64 and channel 4.11, we'll just send you the graph that is for channel 4.11 and nothing else. So as and in the next slide.

D

You can see how the graph exactly looks like so just to answer the question that got asked so and, uh as you can see, we have like uh three four different things. First is the version uh we are currently on version one. So uh there is nothing else there and then the second thing that we have is nodes. The nodes are basically the releases that we have so in this specific example, the release version is 4.11.6 and with that we also have a payload.

D

So this is the URL that the update service sends to your openshift clusters. So let's say: if your cluster wants to go to 4.11.6, it will download the payload from this URL and your plus lexivia knows how to read this so like we just we just look at the graph, that's what the user sees and CBO sees all the internal parts and then we have edges uh as I mentioned. Edges are the like. You can transition from one version to another version.

D

Those versions are connected to each other by edges and we tell that tell it there. So, let's say as uh when. You asked uh previously like if for the skip versions, so all the releases have a metadata like they have like a previous version, so you can update to this uh 4.11.6 from these previous versions. So we just feel there just using those uh in this specific graph and that's how the cluster knows like I can go in this Edge.

D

So that's why we have edges, and then we have this New Concept called conditional edges so like previously, in order to protect our customers, we used to block edges, but now we have become more transparent, with conditional updates. So we'll we have a separate slide to talk about conditional updates, because it's it it it's a very transparent way to tell your customers like this. You can upgrade or not upgrade, or do we recommend it or not. So we'll we'll talk about that in further slides.

C

Protect me if I may add.

B

C

You know how we decide, which is uh like how do we even create the graph you already mentioned the primary metadata. So the way our release process works right. So, if you are releasing version C, we because versions only only knows about version, A and B right. It doesn't know about future it only on over with the past right. So we test those edges through Automation and those are actually built into the metadata.

C

So the when you release the version see the metadata would have yeah you can operate from version a you can upgrade from version B and those things get tested through our release process. The automation uh that has to it must pass those upgrade tests so that we can release it and uh in case there is a change. Maybe we got a bug and we don't want people from a to upgrade directly to C, then what we do is we can?

C

uh No, we don't want anybody to actually directly upgrade from version a to version C, but they have to must upgrade to version B before they go to C. We have we maintain those information in the grab data repository. So we once we add that when they build this release is being built. It actually backed into the my data. Accordingly, the test runs and we release it. So these things are all basically is a big pipeline of tests runs, and you know it's it's fascinating. How we do it. Yeah yeah.

B

And and what in effect you said is if I got it correctly? Let's Let me Give an example which should never happen, but let's say I'm on four eleven one and I am thinking of upgrading to 4116 have not acted, but maybe there are other customers who have found out a problem. They ran into a problem when updating that, if I mean since that data is getting received in the red hat Cloud, we will actually block this mean from getting into a problem. Yeah yeah.

C

Yeah we'll talk about this in the next slides yeah.

D

All right, let's go to the next slide yeah. So now we were talking about how we protect our customers uh from upgrading to a version that that might have bugs. So what we do is we have impact assessments on all the bugs. So let's say if there is a specific bug which might affect the upgrade, so we ask the component teams to help us in assessing the impact of this bug like if it's a update, blocker and if it is what we do is we will just push a conditional update for now.

D

So now we have moved to pushing conditional updates uh before that we used to block our edges. So we'll talk about that in detail like how conditional update solves our problems, but yeah we block it like we do conditional updates in case we find any issues and if it's a candidate release we'll just tune it, so we won't promote it to further channels.

D

um So let's go to the next slide, so conditional update. So uh if you can see on your screen uh like, if we, if you remember the graph, we had a conditional update in the graph that said like there is an issue. So this is how you see it in the UI. So it basically says that uh if you're trying to upgrade from 4.11.0 to 4.11.6 uh upgrading, this cluster is supported but not recommended.

D

And then we have a small snippet which says like why we are saying this and then we have a link uh which will like give you a whole information about what this issue is all about. And if you have like, if they have any workarounds, you can do that or like uh you can get a status update on that link as well. So it depends from case to case, but that's what we do now before this. We used to block our edges and what is to happen is like.

D

Let's say there is a bug which affects vsphere, but by blocking an edge we were blocking the whole open shift like all the Clusters, the Clusters which are not on vsps as well, so that affected customers plans that, um like they tested a single version, they did all the stuff on that version and then Reddit found a bug on vsphere and they blocked that edge. So their whole time got wasted there.

D

So, with conditional updates, we can only block clusters that have issues like I mean we can only stop recommending updates to clusters that have issues and if you have tested everything in your cluster and if you find like this update, is not having an issue in my setup, then you can go and upgrade, uh and you can just take this risk. uh Lala. Do you have anything to add on this.

C

Yeah I mean: can we go back to the previous slide, where we have the graph yeah? If you see in the risk section right, there's a prom query. So what happens? Is the cluster receives this graph and actually the CVR cluster version operator runs the prom quill there and basically, from the result, we figure out whether this particular cluster is actually impacted by the bug or not. So we can actually specifically Target specific platforms, so maybe a bug which is only applicable for ovn right.

C

So now we have a condition there, which only clusters which are ovn would actually respond back and say yeah I'm affected, and that would actually show up in a conditional update.

C

um Previously we used to block it just for everyone so and what is to happen is customers who have spent time in testing a particular version they suddenly see. They have already like taken the maintenance window to do the upgrade and suddenly this is The. Edge is gone from the channel right and that's not helpful to them right. I, understand the pain um and um and they're like we have done enough testing in our environment. We are confident that our production would be fine, um but, and also it was hard for us to communicate them.

C

Why blockchainages? Because of which bug and what is the impact? Now with this um with conditional update, you would be able to see the bug ID, you know, can you have the bug link or do you have a knowledge base article or a knowledge solution? We would actually link it there, so customers can go. You just can go and read the article and understand why we have Tech. We think this is a riskier um thing for that cluster and and for example, in this case right, this is a ovent specific thing.

C

So other clusters, like people, are running open, separency and um they would be getting the available updates. They will not be impacted and uh's question right previously, if you, if you are on 420, 11.1 right and then suddenly, you see there are no edges because we found there's a park, and now you will will not never go to that situation. Where you will not see like nothing, you will always see something.

C

Maybe it might be conditional updated, but you will see something like there is a path, but that's risky in our opinion, so we support it. Red heart supports like red heart. Support is always like it supports you, like uh you know. If you can raise issues with the terms or the CE organization, but we don't recommend it. We think it's risky. You should update. You should wait for another build which has the fix for it.

C

That thinks nothing.

D

Yeah any questions uh with regards to condition updates.

A

No other than surprise I think that's incredible, the the ability to disable edges for a very Target. It makes this service more of an advisor. Much like insights, it is an. It is a service that is intelligent and understands what your cluster configuration looks like and makes advice makes advisement. You know, based on that knowledge, which is really really.

B

Incredible and the more Trump you are, the more acceptable you are yeah. This is this is fantastic yeah.

D

And also we are not like, uh even if we like, so by default. Now we have converted to using conditional updates. So we have stopped locking entire edges, so we had a case where we had to block like all the Clusters, so we use the always case. So in that case uh I mean you are not exactly blocked from upgrading. We are just letting you know that uh you can upgrade, but uh just be careful. uh There are risks associated with it. So we are not stopping anyone from upgrading.

D

We are just helping them to make their decisions in a more transparent way. uh So, let's, let's move on to the next slide, so managing the graph data. So as we talked like how how does a release end up in our graph data so in in your graph? So we do that by using the Cincinnati graph data repository.

D

um It's there on GitHub um I saw gunny post the link, so everyone can go and visit that link. So what happens is when a new release comes out? It will first go so um yeah. It will first go to the candidate channels and it will stay in candidate channels till the Errata is released and once the erata is released. What we do is we move those releases to a fast channel.

D

So if you don't see a candidate Channel moving to a fast Channel, there is probably a bug and that's why we decided to Tombstone it. So we can just stop uh the promotion of a candidate Channel, a candidate release to other channels by tombstone in it, and that's the only place where we like just stop the update from going further for rest of the other cases like for fast and for stable channels. We only use conditional updates now so to transition from a fast to a stable channel.

D

uh We like study the data, so we have CCX folks that help us to make these decisions. So we look at the Telemetry we'll see see if there are any issues for upgrading the Clusters and if there are no issues and if we are confident enough that this specific release can go to the stable Channel. That's when we promote it so that the customer is not affected with bugs, and even if you find any issues after moving it to a stable Channel, we will then conditionally block that update, so that the administrator can see.

D

What's the issue all about uh yeah, like any questions, did you guys see this? We.

A

Had one question I think we're going to touch on in just a second, which was uh does upgrade graph work with uh rhac ACS upgrade automatic upgrade.

A

C

We know um so what what is rhcs stands for? Is it.

A

I'm not sure, let's see I'll bring.

C

It up start talking about the hosted clusters. Yes, they work managed clusters. Yes, they do yeah.

D

And like the Cincinnati protocol that we were talking about uh so Cincinnati protocol is used. If for upgrading the fcos as well is what I know they have a service called zincati, that's the downstream for Cincinnati. So we do have such services in Red, Hat uh I'm, not aware of all of those, but yeah uh like there are things will be similar, like the concepts will be similar in the services as well uh yeah. Let's, let's move on to the next slide, so upgrades and disconnected environment.

D

Now you will be thinking like what happens if I cannot reach API dot openshift.com if I have a Sandbox cluster, what should I do so? We have a solution for that as well. We have a openshift update service operator, which you can host locally in your cluster. Doesn't matter, uh I mean you can host it on a disconnected cluster or on a connected cluster? If you don't want to reach uh Red Hats services so and the possibilities are unlimited. So, for example, you have a like a 100 clusters that you want to manage.

D

You can have your own disconnected. You can have your own update service operator and manage the upgrades for the Clusters to that operator and you don't have to rely on API dot openshift.com. So we have that and just to update yeah we, the um versus works with rhac ACS as well uh so I think that answers the question. uh Yeah.

A

D

Like for uh in disconnected environments as well for the operator uh we have like, we depend on like.

D

Yeah yeah yeah so like we have like two dependencies. If you wanna run the update service operator in your cluster locally, the first one is: you should have a graph data container so because you cannot reach the graph data repository on GitHub, uh you need to have a container, that's basically uh the graph data repository um locally in your disconnected environment, and then you need to also mirror these images into your container repository that you can reach. So people can do that using OC mirror.

D

We have a tool in openshift, so it helps you mirror the releases from openshift to your container repository. So you don't have to do that uh manually a lot. uh So, let's, let's go on the next slide and let's see audio Yeah, so the osus operator, you can find it on the operator Hub. So it's it's! It's easy to install.

D

You just need to click on install just like you do like any app on your phones and it will install the operator on your cluster then, after that you just need to create a CR with, as I talked the graph data, the graph data container and the release images repository, and that's it that's the setup. You need to run the openshift update service in your disconnected environment, and that should do it just work so best practices. Lala. Do you want to date.

C

Yeah, so we already little over time so I quickly go through this one um I. We suggest you to upgrade frequently, especially with the G stream updates, because for each job openshift we fix more than maybe 50 bucks or so um so we are fixing a lot of bucks so uh and also the the reason why I'm suggesting frequent update frequency would work better, because uh you know when you have a less Delta code changes applying to your cluster.

C

There is a less chance of getting some trusting failure, even if, with all our efforts, there's still, you know, no code is Puck free and there's a possibility. It's always a bug somewhere else which is not tested right so and with gstream updates. You can actually pause your worker polls and can just upgrade your control planes. There are documentation in the openshift docs docs.opensure.com, um making sure uh when you before you upgrade, you can check, get lost operators.

C

They are not degraded because if they degraded there is a symptom Uh something's going wrong, maybe not completely wrong, but some warning kind of symptoms and you should wait for it to go away, use port disruption, budget for a workloads, because we respect that.

C

So you have website for disruption project your you know this extra layer of protection that your application will not break during upgrade and uh if you do, cannot take a peak maintenance window and you want to do small chunks of upgrade every each time because you want to you know, don't want to take the risk of operating all the notes. It's in one go. You can actually use multiple machine, config pools and then upgrade as whenever you, whenever it's convenient for you to slowly update the whole cluster.

C

So there's the information out there in docs.opensee.com, we are still up we're trying to improve our documentation to communicate more yeah. Next to the next slide.

D

Yeah so next we have a demo.

A

Probably we'll we'll leave that as a teaser for next time, so we we can. uh We can end the stream uh in a little just a few minutes, um but I I know there were a few other questions on a few wonderful things. I wanted to say I these. These folks sent me some links beforehand. I've been if you hear the leaf blowers outside I apologize they've arrived, but I uh I've sent those links in chat.

A

So if you're interested there's a bunch of links for some of the interesting, um you know the openshift update upgrade graph and the Cincinnati graph and some of those other components all open source all out there. Here's those links here I've been copying and pasting these links as well.

A

So, if you're interested in accessing them- or you want any more detail after the stream, you can always reach us at Cloud multiplier, redhat.com and I can I'm able to connect you directly with Lala and with critique and they'll, be able to help fill in those gaps as well um anything else, folks for guys before we we close things out.

A

If not, we can go ahead and close out the stream, except for one thing, which is to say our next stream is going to be talking about Disaster Recovery in ACM and a multi-cluster level. So a really good follow-up insights, giving you insight into your clusters, update graph, creeping you safe through upgrades and finally Disaster Recovery to help you when. Well, maybe the data center is underwater.

D

A

Graph can't help you if the data center is underwater, but maybe Disaster Recovery can't so that holistic story hopefully will have a good answer next week. As always, thanks for joining us, and thanks for this amazing presentation and all the work you do keeping clusters stable and reliable and upgraded, and as always, we will close with the intro, the intro as the outro and see everyone in two weeks, good.

C

Talking to you.

C