Red Hat OpenShift The Level Up Hour | Red Hat Livestreaming, 30 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Level Up Hour (E39): Connected Customer Experience

Description

Ever wonder what happens to the health data of a connected cluster and why it's a good idea to have it enabled? On today's episode we talk to Jan Zeleny and Ivan Necas from Red Hat's Connected Customer Experience engineering team to hear about specific use-cases for telemetry and insights data, including user-facing applications, support enablement, as well as analytics work that helps the OpenShift engineering team to build better products. We might also talk about how that work relates to the role of the SRE team, AI/ML or even COVID-19: everything is interconnected.

Follow us:
Langdon White https://twitter.com/1angdon
Chris Short https://twitter.com/ChrisShort

What is the Level Up Hour?
Every Wednesday at 9am ET we stream live demos on how Red Hat Enterprise Linux (RHEL) users and admins can benefit their organizations and improve their careers by learning how to use containers, Kubernetes, and Red Hat OpenShift.
Learn more at:
https://red.ht/leveluphour
http://red.ht/3ayHlHI

How to participate in the The Level Up Hour:
Watch on YouTube every Wednesday at 9amET
Chat with us on Discord https://discord.gg/5VMVGJt
Check out our GitHub repo https://github.com/level-up-hour/

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
https://www.openshift.com/
https://github.com/openshift/

#RedHat #OpenShift #tluh

A

A

A

A

Good morning, good afternoon, good evening, wherever you're hailing from welcome to another edition of the level up hour, I guess I should say episode these days, but another episode of the level up hour. I am chris short executive producer of openshift tv cncf, ambassador to the stars, et cetera, et cetera, I'm joined by the illustrious langdon whites and two of my fellow red hatters today langdon, what's going on. Why are there so many people on the call exactly.

B

Just us, you could also you could you could start using uh chapter and just character, yeah.

A

um This chapter of this episode is.

B

Right right exactly uh yeah, so this is the level of power where we talk about containers and why uh you know, like they ain't just hype. You know, there's uh there's a lot of usefulness to them uh and to try to talk about why they might be useful to you as like a rel administrator or a developer or whatever, even if you're, not using them necessarily in production, um even if you're not using them in an orchestrated fashion. Just kind of to show you why containers are interesting and what they can.

B

Let you do um you know I. I run a a series of points on the show, uh a lot of the time if you've ever seen the show before we'll talk about it later, but um you know even for that. I use container to run that, so I don't have to run like a google's sdk in my laptop. uh So as just a quick example, we've done a ton of different episodes. I think this is episode like 38, maybe um 39, where chapter 39 is maybe we'll start calling it um but today.

B

uh Well, let's, let's do the slides and introduce the uh the folks for the show, um and uh that way we can make things less confusing.

B

All right, so everybody should see my coolio slides. As always, uh all all discredit goes to me for the quality of the uh slides that are are produced, um but these nice little art things over here are pretty cool um all right, so you can find us on twitter. I'm langdon with a one uh and chris short is chris short, with a c and an s and another s, and then some r's et cetera, you get it um and uh you can also find us on our discord uh check it out.

B

You can, you know, come and hassle us about uh what you see on the show what you might like to see on the show. uh What other shows you might like to see, uh or any other kind of random questions about containers and orchestration?

B

um You know you could also ask this about dating, but I highly recommend no.

A

B

That, um but uh you know not at all, we are. We are both married, so there's there's something that we're doing right now: yeah.

A

B

Don't know spam is real yeah exactly so, um but, like I said, we do try to help out, uh particularly on our discord. uh There's a community of people there who might also be able to help with kind of whatever questions, or you know theories you might have um moving on uh today's episode. I got that number completely wrong.

B

Clearly, wow um is episode, 39, not 37., uh and so this is where uh we'll talk about this in a few minutes, but the you know every time we have a guest on the show uh who's from red hat. We talk about red hat, renames uh teams and uh titles and stuff. Like that, all the time it gets very confusing uh internally, we generally speaking, try not to expose that to the external world, but sometimes it slips out.

B

So today we have a couple of guests and I'm going to let them introduce themselves and what exactly their team is called, because, even if I knew it last week, I may not know it this week. Show notes are not live, uh as I was mentioning to uh chris earlier. uh I've been on vacation for a few days, uh so I didn't quite get to them. uh So I will tweet them out as soon as they're, ready.

A

I'll link to the main repo in chat for everybody.

B

Oh, thank you yeah, so they will show up there uh as soon as I get them done, and actually I will also fix the bug that was reported me on twitter, which is uh fixing the table of contents, uh which I haven't quite figured out how to automate. So uh that means it relies on my memory to do it. So good luck, uh yeah! So without further ado, um why don't we start with yon?

B

uh Can you kind of introduce yourself tell us what your role is and uh you know, and the team you're working on.

C

Sure thing, uh but before I do that, uh can you send me a link to those cool little graphics that you had on these slides yeah.

B

They're mine, they're, all mine,.

C

uh All right yeah, so my name is sean zelani, I'm uh I'm the manager of the connected customer experience. uh That's the internal uh name of the team. uh Surprisingly enough that team kind of stuck for the last couple of years, uh so we might be renaming the team soon right.

B

Right right any day now.

C

Yeah yeah exactly and uh well originally we started the team with like a broad vision of like hey. You know we have. We have a bunch of connected clusters here. Let's do something with it something useful. uh Let's present uh some value back to the customers, and you know, let's present some value to our engineer, to our engineers internally. uh It wasn't exactly clearer than that. um Since then, we actually have a mission now uh and let's see if I memorized it, uh people asked me so many times. I think our mission.

A

Command yeah exactly.

C

Yeah, maybe next time yeah. Our mission statement is to improve the operational experience of connected customers and connected clusters. uh Whatever that means uh we we do that on so many fronts. uh We work with our support engineers. We work with our product engineers to improve the quality of the product uh and obviously you know we present some value back to the customers, so they are actually motivated so that they can actually get some value uh for sending us uh their telemetry data.

C

So that's that's in a nutshell. I guess.

B

Cool um all right, and so now this one's hard, because uh I've never heard your name out loud. I can do it, so I can do it, but it depends on. It depends on your nationality, which I'm not sure of so it could be ivan or yvonne or um go ahead. It's ivan.

A

B

Nice nicely done yeah um yeah.

D

The the surname was was right, the the first name, uh langdon, had it better like with evander, but you.

C

D

I'm really used to to all of that. So don't worry about it! So yeah, I'm.

B

Software engineering and working clear you know.

D

B

Me about it right, like you know, I don't know.

D

How people look.

B

At it and figure out all different ways to pronounce it, but it's very impressive.

D

Yeah so yeah, I'm a software engineer working on the same team that diane also connected customer experience. I officially or not. Officially I have the architect name uh in the role, even though I don't like this particular name, because it just has a lot of assumptions. People have about this particular role and basically, I'm trying just to to figure out like what would be the most important or interesting thing to do about our data and bring the impact back to our customers eventually.

D

So that's in a nutshell, what I'm trying to do and then trying to get involve everyone that tried or makes sense uh in this mission.

B

Cool all right, so obvious question: what does connected customer mean or or let's put it slightly differently, um you know what is the external facing like what is what are people who are our customers or users or whatever? What do they see that your team produces.

C

So let me answer you answer that with a question, have you ever heard of uh red hat insights.

B

Yes, I have in fact we actually had. Oh no, I don't remember. We had somebody on the show talking about it. We had uh it was episode, 36 or 37 yeah.

A

Yeah, so uh we had.

B

We had gerito on talking about subscriptions, which was is kind of getting morphed into that, and then we also we did something with insights as well. um Yeah we've actually done a few different things, so yeah. So in a nutshell, this is the way I understand uh redhead insights, it's like I am gonna run. um You know it's gonna, keep track for me of all my rel uh instances and kind of their current status and whether they're in good shape or not, uh is kind of what I said in a nutshell.

C

uh Yeah, that's definitely that's definitely true on the uh on the well side, uh but you know lately. Let's, let's imagine insights is a big, overarching brand and not just real specific, but you know like uh openshift and in the future, uh perhaps ansible as well, uh but you know ccx and what we do on the customer facing side. You know is the first step for red hat to you know into the insights world on the openshift side. So uh what we?

C

What we brought last year, was a well you can call it like a tiny application. It's not that tiny! Actually, uh but it's the insights advisor for openshift, which uh you know it doesn't give you all the information you will see in rel, because in reality you have advisor. You have vulnerabilities compliance all that stuff.

C

On the openshift side we have just advisor which gives you some recommendations about. You know how to keep your cluster healthy. uh You know to put it in in one sentence: we look for different types of recommendations related to stability, availability, performance uh security, sound. We have some security guidance as well. So you know that's what we have on the on the customer facing side. uh If you, if you want to see that I can actually show how it looks like sure, yeah, all right, okay. So let me pictures.

B

Make it easier to understand things a lot of time, yeah.

C

uh I think so too, all right, so uh I hope you can see my screen now. uh So this is what you will see as a as a cluster admin, either either this web console right here uh or you can see you can have, uh you can have obviously command line access. uh You see that yeah, I'm logged into my server. So this is this. This is the regular stuff right and here you can already see the first glimpse of the insights advisor for openshift.

C

uh You can see this is I installed this cluster just yesterday, so you can see that it is perfectly healthy if you, if you want to take a look at it in the openshift cluster manager, which is another nice app that we have on cloud.reda.com, it will show you some some details about a cluster. It seems that I'm logged in already.

C

So this is the overview of my cluster. uh You can see multiple tabs insights advisor is one of them. Again. You can see that you know everything. Everything is in a good shape, but what would what would you see if things weren't in a good shape? I actually prepared a short little demo, uh where I'm gonna, where I'm gonna show uh some. You know. How does it look like you know what insights does for you as as openshift admin, so you're introducing problems is what you're saying yes.

B

C

Intentionally going to introduce, uh introduce a problem here and then we'll see what recommendations we get and how can we? How can we follow the recommendations so that the problems are fixed?

C

So, let's start with creating a couple new projects, I'm gonna create project and say project copy.

C

So a couple projects here and uh you know I'm gonna skip. I'm gonna skip a few steps now, because, obviously I would now create applications within those projects. You know put some code in and whatnot, uh but just to just for the sake of the demo, uh I'm gonna jump to almost the end and send some egress ips for the project.

C

D

C

Have this somewhere in my history, I.

B

Hope the number of demos I've seen that are purely about the usage of bash history is uh very impressive. Right.

C

Yeah uh no surprise there: uh okay, so I'm gonna patch net, the net name space for the project and set the egress ip. Let's keep 100 here so until now everything everything is okay. This is not gonna do any well. Actually, this is gonna, introduce one one little thing already, uh but let me introduce uh another another problem: I'm gonna keep it keep the 100 here and let's say I want to create an egress ip for my copy project, and I kind of forget to you know: change the ip address here right.

C

So now I have two projects with the same egress ip. That can't be good but well it is possible that I kind of missed that for whatever reason uh so now, when I take a look at that.

B

C

uh Just to check that everything yep so two projects with the same egress ip uh now insights uh has a component collecting the data on on openshift and sending that data back every couple hours. That component is called insights operator and since it collects the data, only a couple hours you're going to see me, uh you know forcing the data upload a couple times here and I'm going to do that by going to project inside open shift.

C

C

uh Okay, oh see, and I'm gonna, I'm gonna delete this spot.

C

Which is gonna, the inside operator is gonna, restart the pot and, as part of the restart it's gonna. It's gonna upload the data. So uh in a little while we'll probably see here that insights is not available, which means we're on a good track, uh but the refresh takes a little while so I'm gonna switch to the openshift cluster manager uh refresh the page to see if data was already uploaded and processed.

B

So when you say openshift cluster manager yep, so that's a single cluster you're, looking at all right, uh yeah, yeah, yeah.

A

Okay, yeah, for some reason, I I hear.

B

That and I often think of uh acm like multi-cluster manager. I don't know why, um but they get completed in my head very easily.

A

It's not like, we have a lot of acronyms floating.

B

Around no no right, um but at least we have it was funny uh somebody made a joke like that yesterday and uh and they rattled off the a very long-winded title whatever- and I was like. Oh you forgot, red hat has to be in the middle of it somewhere.

C

Of course yeah I mean you can see that you have. You have a bunch of clusters here in in ocm, but uh you know the functionality that we have with with insights at the moment is just a single cluster view.

C

So that's that's why I use this. We use this view, but you can already see that the data- I I you know I uploaded yeah, you know- was uploaded and processed. uh You can see that the last check happened just now or one minute ago. uh Let's see insights on the cluster. This is the web console on the cluster inside. Here is not available here, meaning uh you know it's gonna refresh any in a minute now and I'm gonna see pretty much the same information here.

C

So, let's, let's go back to this one uh in a little while and let's take a look at the recommendations, this is the. This is the first piece of value that you get uh from uh from insights advisor.

C

You can see that by doing what I did, I actually broke two things uh and both related to cluster dropping some traffic, uh the first one, because two net namespaces contained the same address ip yeah. We got we kind of knew that was gonna happen, but there is another one.

C

None of the egress ivs I I set to the projects is assigned to any node, so you can go. You can go here. Take a look at the details. It will actually give you specific information about. What's going on, you can see the names of the projects that are or the net name spaces that are problematic. You can see the ips that were set and it gives you some some basic guidance and there is usually a link for more information like this one, which will take you to a openshift documentation about setting setting egress ids.

C

So, let's, let's try fixing uh fixing this documentation alone right, um uh no, this one uh so that so the duplicate uh duplicate, egress ips. So now uh you know. If I saw that recommendations. What I would probably do is to take a look at the namespaces like I did the first time, and now I see that you know exactly what the recommendation told me and okay, so I I need to change my ip for this project.

C

So little uh little bash history magic again and let's use uh 101.

C

Quickly, quickly check: what does it look like right? This should be now good and again. uh I would need to wait for a couple hours for another data re-upload, but I can I can force it again.

B

How often, how often does it kind of run by itself.

C

Every couple hours like.

B

C

Once every two hours.

B

Yeah, so that's I mean that's an important thing to note right is that you know if you're doing a whole bunch of operational stuff right. It's probably a good idea to kind of force a run. You know it might be worth considering you know. Is there a way to kind of trigger it without having to delete the operator.

C

um That's a good tip for the future.

B

Yeah, just because you know like you tend to you, tend to go in and do a bunch of those bulk things at once. Right um and then you know, and it's like you, you need immediate feedback on that. Not you know not. Two hours later, uh when you've already gone off to get you know a sandwich. Yeah.

C

Yeah, that's that's a good point. We should. We should definitely look into that one and so yeah. Now you can see that uh we're one recommendation down. uh One remaining uh this data on the cluster hasn't uh hasn't updated. Yet so you can see still the old one ah not available. So it's refreshing, that's good!

C

uh So in a minute I will, I would see the same information as in the openshift cluster manager, uh just what just one recommendation. So, let's take a look at this recommendation.

C

uh Okay, simple explanation: let's see what are the remediation steps.

C

Namespaces I get linked to documentation yeah, you can recipes for the project and it includes some information about the yeah.

B

C

See some information about the host subnet, so that's that's pretty much what I'm, what I'm gonna do now, uh so I'm gonna fix this uh fix this issue by following the recommendation and setting some setting the egress ips for assigning them to some host subnets.

C

So quickly checking those subnets.

C

You can see here that no egress ips are set for any of the host subnets. So let's, let's quickly check what are the uh okay, I'm gonna pick pick this worker note. That's this host submit.

C

All right, then, I'm gonna set uh basic, oh all, right, different addresses here. uh You know I can. I can either do it. This way set some specific, fixed, eagles ip to the host subnet or I could do like a address range, uh whatever whatever I prefer in this case, I'm going to do two specific two specific addresses and check that this applied yeah egress ips both of the crease ips are here. So you know the drill oc, get pot, delete.

C

C

All right going back to recommendations, see how how fast we are. Okay, not that not that fast, I still hasn't uploaded.

C

Let's do a refresh all right, so uh you can see we're back to all cluster past your cluster pastel recommendations, meaning uh we don't know about any known issues that would be that would be affecting uh your cluster and again, insights is not available here. It will be reflected in a in a little while.

C

So that's that's pretty much it that's that's in, and actually you know how you, uh how you consume this stuff and how you how you get the value out of out of the insights advisor.

A

So I'm going to ask the hard question: what data do we collect.

C

uh Yeah, that's a good question.

A

Obviously I mean I have some docs that can help, but yeah yeah.

C

Yeah yeah, obviously we have. We have a documentation on that, because you know the data, the data set changes and it changes with with every version, and sometimes we even backboard, some of the some of the data collection code and so uh yeah. I would I would point you to if you, if you search openshift documentation and you search the if you look for section remote health, it gives you an overview of what data do we uh do.

C

We collect, uh I mean, and we we also have if you, if you want to look at the code of insights operator, obviously it's open source. uh So if you go to our github repo, which is in in the openshift org, the repo is insights operator, we actually have a sample archive which, uh which shows you like. How does the data structure look uh and that will give you a good idea of what we collect.

B

So before we move on um so someone who we refer to as netherlands, hackham um made a joke about uh rarely ever getting to zero uh issues on their cluster, um and so, while we're kind of talking- and you know, because the lag sorry um if he could uh mention a couple of things in the in the chat uh that he's seen before you know that uh he doesn't feel like they can get fixed or whatever. That might be a useful thing to talk about.

B

C

A good point, one one thing about our recommendations: uh you know we are. There is a difference between insights, recommendations and alerts. uh Like you will see alerts in your cluster, you know whenever something is out of ordinary, but sometimes or you know many times, these alerts are not exactly actionable, but they will. You know, there's some indication that something's going on, whereas insights recommendations, we publish recommendation only when we are when we know specifically what was going on. We can detect the conditions and we can also give you a recommendation.

C

You know what to do about the situation, which alerts typically uh typically, don't have so you will see that there is much. There is many more alerts that in than insights recommendations, so it is not uncommon uh for a cluster admin to see a long list of alerts. While insight says, like you know, everything is quote. Unquote: okay,.

B

uh Is there one somewhere where it just says, light it on fire and go get a coffee? Because because I I think the temptation would be very, very high for me to put that in there somewhere.

C

I think at that, at that point inside just you know, hides and hides and runs away, and you will see your insights is not available.

B

So um you know so we go back to the connected part right. So so, what's the point of that like what do you like? Why is the is the cluster connected uh and what does that mean.

D

A

D

Yeah, I I might try uh so the the basic idea was with the openshift four, where you know in like openshift, three and and before it was always a bit of a guesswork of like what problems uh users actually run into.

D

You know we we have the ci system and testing and qa department and all you know, ticketing, customer cases and so on, but it still doesn't give you the good picture of like what are the real problems that the customers customers have and at the same time you know like, especially when, comparing to marriage services and cloud services, uh there is this deficiency of not being able to know like what what problems the customers.

C

D

Was quite high and you know people decided we need to do something about that and that's where the idea of actually making the customers and clusters connected which by this means mean the customers sending the health data about uh about them or their health data to the cloud and to callrail.com to be more specific, where we have pipelines and processes to actually handle those data and to be able to uh do better decisions in terms of prioritization of the fixes, and things like that.

D

So basically, the main part here of what the connected cluster is is the cluster that is sharing the health data with with red hat for the further analyzes. So uh there are two components uh providing this capability. We already talked about the insights operator as being one of those. uh It would be mistake to not mention the second one, which is the promise you slash telemeter project that is sending a selected set of metrics to uh to uh red hat as well, and we have a way how to uh then do the aggregations.

D

On top of this data to be able to actually help the customers either. At the specific, like single single cluster level- and there are use cases for the support itself to be able to leverage this data without having to you know every time ask for, we must gather to to understand what the issue might be, but it's also for the more fleet-wide views where we we can know what what troubles are our customers hitting in order so that you can do their life better by no doing better releases next time?

D

So that's basically another use case. Besides just doing the you know, direct recommendations to the customers where the customer still uh benefit from from this data, even though they they might not see this, as you know, direct recommendation for them. So, for example, uh like it's quite connected also to how we do the upgrade routes for openshift4, where, when we first do the release, it doesn't go or is not available for every clusters.

D

We have this fast and candidate channels and compound to this, like when we release a new version, we are still monitoring even monitoring closer the clusters that go through this upgrade and see. You know what issues they are hitting because, even though we try our best, we do our best in terms of uh making sure that the things that we could think of don't break, there's still a chance that something that is specific to customers environment might make some uh issues more obvious than what we could ever think.

D

So this is basically one of one of the cases where, before we roll out the upgrade to the wider uh wide range of customers, uh we are still watching like what's going on with the smaller set and things like that,.

C

So, by the way, this is how this is, how I explain you know: what's the difference between connected customer experience and kiwi like it's.

D

The job is the job of tv.

C

To you know make sure that, like no issues go into production, you know in the ideal world. Obviously, while you know it's, this is not our job. Our job is to monitor. Like hey. This happened once there is a chance that it's going to happen again, and you know if it happens, uh you know frequently enough. uh We we need to work fast to make sure that it doesn't impact. You know like a big portion of the fleet.

B

Well, it also, I mean one of the things that you're capturing right. That qe, you know, is never going to capture unless you actually change the development model, which is that operator introduced issues as well right. So so it's not just that the cluster is in bad shape because of the upgrade right. It's because it may be that you know in openshift 4.6.

B

You had these two things, you know, and they were okay.

B

You know or weren't causing any problems, but then, when you upgraded to four eight, um you know now you know that that back ported patch, that you were relying on or or sorry that's the wrong way, but like that that bug you were relying on it, didn't necessarily realize it you've now introduced it right, like my classic example of this right is when uh I think was: windows 3-1 no might have been later might have been windows 98, but when it came out um civilization, the video game wouldn't run civilization.

B

Three, I think, um and so, and the reason is because they were relying on a bug in the os to do something for performance reasons right which is pretty typical for video game companies, so microsoft actually reintroduced the bug into windows to make civilization work. So you know it goes both ways, but the point is that um actually I I almost failed a test actually like a certification test, because the documentation said that this thing worked.

B

One way- and I know from using it that a couple of those things did not work uh because there were bugs, and if you try to use them, things would go badly and I had never read the documentation right.

B

I had just done the actual thing, um so you know there's a lot of those right where you kind of as a developer and operator or whatever there's some issue or something that you have a workaround for, or you know that you even don't know is a bug, and these are the kinds of things right that something like insights is going to catch, but kiwi never will right, they just yeah. They don't have the data. You know. I just showed.

C

You one of those life.

B

Yeah yeah exactly.

C

B

A

Was I mean so, but.

B

That one's obvious right, like as in you know, that's just a straight up error right, but there's also these other weirdness things that'll happen because you know like I said, because a bag bug was patched or because you know the way something internal is working, move to a slightly different model. You know so it could be pretty subtle. You know more than just you know,.

D

And I have actually an example of that. I prepared one one of the demos or like uh showcases, of how this data actually being treated. So let me try to share uh my screen so as I.

A

Mentioned the monitoring.

D

Of the upgrades uh and and looking at how uh it works, it's one one of the things that uh we are looking into so, for example like when we are like it's at some uh point in time like this is not the fresh data, but something that I had a few months back when we are looking at like how long the upgrades take uh between different uh y stream versions, we we've seen that for for 4.7, there is more kind of variety of of the of the length of the upgrade.

D

So you know you could you could see this as the the length of upgrades uh in hours? I think, and we see that you know there's something going on in there. So the next thing you are actually interested in like what are the common problems. You know common reasons why you know out of the salon in photo seven upgrades that uh you know longer than before.

D

So what we did uh as one of the views is basically looking at the some uh common causes for for clusters or upgrades to taking longer which, especially in this sphere, uh we got to uh this symptom, uh which is the connected to the sphere problem.

D

Detector, I don't have the you know more drill down uh example in here, but basically the gist of that was that many clusters in 4.6 had a problem with uh in vsphere configure configuration where, even though the cluster thought that no the psp is managed- and it has the right credentials to do no machine, config operators and all these kind of underlying infrastructure operations uh like on behalf of the administrator in reality, the credentials actually were not set correctly because the users actually were not using this functionality.

D

But you know at that point. The cluster itself actually was not doing any checks that know the credentials whether they are correct or not, and when the photo seven went out, this check was introduced and what it cost is basically once upgrading to 4.7 the upgrade. Would you know basically stop at some point? Luckily, it was at the end, but it's still marked as unhealthy state where it was complaining. You know justifiably that the credentials are not there.

D

So no there's something wrong with configuration from and no I can imagine winning this conversation over qe that no this is expected state. We want to let people know that uh something is wrong.

D

The problem is that from the user perspective or customer perspective, the cluster was fine in 4.6, you know, and they were actually not using this this this functionality and the only thing that they noticed that the upgrade might got stuck eventually when, when they don't have this configure, so it took some time to you know, have this discussion with our engineering to actually uh find better ways how to let people know that something is not not right, but still not marking the operator is degraded, because that was just too intrusive to the to the customer and their workflow.

D

So that's. You know some of the example that shows a like how we are actually monitoring the upgrades and looking at the common causes for aggregate failures, but it's also, you know the difference between how one can look at this from the perspective of the engineering and quality assurance, and what's the perspective when looking at like what the customer is actually seeing and experiencing. So uh that's, I think you know, covers one one of those similar examples to what you mentioned uh with the bug.

D

In windows, where no, we need to work with what people were used to, and uh actually not being too aggressive on fixing things that that might be working for them. Let's believe in this way,.

B

Yeah yeah, it's a it's a weird, it's a weird fine line um and to be clear right, the window story. I did not hear it from a windows engineer who introduced the bug. So for all I know it's a pocketful, but I think it's a great story. um So, uh let's see what was I gonna say? Oh so uh and jan knows where this is going, um what how do you figure out what the rules are?

B

um So in other words, you know, are you you know you, your team, you know other people, whatever uh sitting down with the data and kind of looking for issues and then looking for recommendations or as like, I said, jan knows, what's coming, um you know. Are you introducing some ai here some machine learning type models to try to predict what kinds of errors you might be getting.

C

So I'm gonna start with the with the simple ones. uh I would say it's, it's actually. All of those combined uh like you know the way. The way we treat rules is that you know it's. It's encoded knowledge of our support and product engineers. You know that's you know that that's the value of openshift uh compared to kubernetes same as it is, for uh you know, rel compared to centos, for example, because we literally encode our knowledge into those into those rules. So we have nominations coming from support engineers.

C

You know when they see a frequently occurring uh occurring, support cases. You know they nominate them for automation, uh I mean you know they themselves use internal deployment of of insights advisor, which you could see of internal deployment where they have many more rules than what the customers see. You know some of the rules are not as descriptive, but they still help support it right. That's our one source- uh and you know there are multiple other people who engage, engage with customers and then bring back the knowledge and and give us tips like hey.

C

We should we should get this recommendation or that recommendation. So that's the that's like the trivial source of you know denominations, but then.

B

I wouldn't characterize it as trivial. That's that's actually very hard data to get, but it is. It is simple. Perhaps.

C

Yeah yeah it it is simple in, like you know, non-technical for you know, for us engineers, uh you don't need to code to get that information, sometimes that's harder than coding, but you know whatever yeah right, uh but that yeah. Then then we have other sources and and I'm gonna, let yvonne talk about those. Those other sources, like you said, like some analytics and whatnot.

D

Yeah so- and you know we might getting more into the ai and ml stuff, and one thing that, like me personally and many people, especially on like like having more practical experience with the underlying technologies, they like, we are usually a bit more skeptical on seeing ai and ml, like solving all the all the problems. You know just like that, and that's actually confirms also our experience so far where the the ml part is.

D

Actually you know the cherry at the top of the all the work that needs to be done before it can be applied, so uh would be, and we still want to leverage the data that are that are coming. But there is a long path before you know, like from the raw data to something that can be plugged into the machine, learning models and benefit uh platform. So the first thing that we need to do is actually cleaning the data and having the right views.

D

So uh it's like, even though some of these data might be clear, but with pronq ul people start realizing that, uh like running from qr on on larger data sets- and you know, if you have too many time series uh coming in uh it's- not that the trivial as if, if one had a single cluster deployment and promise use on top of single cluster. So uh one thing that needs to be overcome is basically finding a different way to store this data and allow clearing those so that you have the right views.

D

So once you have that you can also starting getting more people involved outside of just the data analytics itself. So we try like in our team, we have a dedicated team that is more focused on looking at this data and trends and and trying to do no recommendations for recommendations. For example, no finding like what would be the best advice is to ch choose and actually refine, so that more people would be helped with that.

D

But even even better is to get these views to the people that actually understand the openshift better than we are, which are either engineers in openshift core or those are the support engineers working on the on the on the cases or our sre engineers working on operation dedicated and if we are able to get their eyes. Looking at the data that we have in some understandable way, that's in my experience more beneficial than trying to apply some machine learning models right away and finding some.

D

You know, nice information that without any interpretation still this doesn't give you any value. So we at some point, I think, are even acting like uh translators between the engineers working on on openshift and data scientists that we have within the red hat that are working and experts on working with the data. So one thing that, for example, they have different language or even worse, they use the same words for completely different things. So you can think about like if you tell a cluster or uh the other example, a feature.

D

If you talk about clusters and features and openshift engineer, they, you know see some new buttons uh to be added to the the deployments. When you talk about clusters and features with the data scientists, they see, you know the clustering, algorithms and selecting parameters that you fit in, and you know this is a simple example, but uh there are many many of those where, like there needs to be effort, put into and understanding both worlds, so actually the that benefits can uh be derived at the end.

D

So that's something that uh we try to kind of help and uh basically.

B

D

These different teams working together uh and getting the benefit out of the data, so uh I have some examples as well on you know specific things where we are applying the more advanced stuff from you could even call it machine learning. uh So I might share the screen again.

B

The um one of the things I am most proud of that red hat is uh introducing the phrase. uh Words are hard at red hat um and, by extension, there's also. Words are hard in english, uh which is this problem right. Where we have a lot of synonyms. You know, there's a lot of jargon and when you mix fields together, you know those jargon. You know conflicts really start to show up, um but yeah. That's that's interesting. I hadn't really thought about the feature.

B

The word feature, because I I've also played in both sides of that you know, aiml um data science. You know a feature means one thing uh in software engineering a feature means something completely different, uh which is kind of entertaining I'm sure.

D

Yeah, so what we are looking at is some views at like we call it symptoms because you can have either you know, alerts that the the clusters are sharing or you can have degraded operators uh and different reasons for that you can have the insights rules. You know things like we've seen with egress that also the clusters are reporting. So we unify these these different data points and we call it symptoms and assign them them some, some id that we can work further with.

D

So in this particular case, it was the monitoring operator marked as degraded, and what you can see at this graph is some spikes. So this on the x-axis, we see the different uh z string versions for the open shift and on the y axis, it's some hit rate in terms of you know how many clusters uh are seeing this particular problem in in particular release. So what like?

D

So? There is a lot of work to be done in order to get this view. You know, you know if you have the raw from ql, it's always uh on x axis. There is a time so no switching it in a way that you are not looking at the time series from the times perspective, but on the time series in terms of different versions is some work to be done, and somebody needs to the second thing either.

D

Looking at this and giving this in front of the eyes of the subject matter, experts or uh if the, if the data allow that we can try to do some auto detection of this thing. So you know it seems, at least in this example, that it should not be that hard to define this kind of spikes or signals within the noise that we should actually notice early and act before uh it starts causing problems for for many many more users.

D

So uh here is some visualization of like how it looks uh in in real world where, with the red line, is basically defining some thresholds based on the previous history of the particular trend, and if it crosses some some boundaries, then we mark it as anomaly and something that people should be notified and start looking into more details about the root cause of the problem. We still know it might be an alert or degraded operator.

D

There might be more kind of root causes for that, so it still takes some time uh after the algorithms notice that, but uh it's the first step being able to to notice this quickly. So even this is not that obvious.

D

You know it's obvious in this example, but when we actually start looking at it or applying these algorithms against every single symptom that we observe in openshift, you start noticing some things that are triggering the algorithms to you know to send a notification, but in reality it's it's not not problem and one needs to have some understanding of how openshift works, and you know uh how was the behavior of, for example, alerts when the installation or upgrades is happening before one actually applied it.

D

So there's still more work on the cleanup side than on the machine learning algorithms themselves. So that's something that I want to. I I don't know how much stress I would put on this- that the data cleaning and making sure that makes sense it's much more important than tweaking the algorithms and applying the latest machine learning neural networks. On top of the data, because, that's that's usually even simpler, regressions might work quite well.

D

So, uh for example, in this case of uh having the spike detection, uh what was causing troubles is uh we tried to note this, like whenever there's a new release, charting multi symptoms that have higher heat rate that what we were used to in the past and one one of the things that happens? Is that usually like?

D

There are two ways to get to the specific version of of the open shift it's either via installation or via upgrade, and both of these processes include some intermittent alerts to be triggered because you know modes are getting restarted or you know it's just part of the installation and upgrade process that things are not great at all at once. You know atomically, it's still some some time where things might not be as expected, and that usually causes some others to be triggered.

D

So when looking at the data from this perspective, we always got quite high hit rates on the on the fresh release when it got rolled out and in order to clean this up before applying the machine learning, you need to exclude the times when the upgrades were happening, or you know you can focus on symptoms that are happening more than one day for one cluster and they're having the trends from that which eliminates some of the noise, and then you get much reliable source of data to be able to to do this kind of analysis.

D

On top of so it's really interesting work and you know a lot of detective work as well. So it's not just about looking at the data, but we many times need to dig deeper into like what the particle problem means. You know what are dimensions of of these within the box with our customer cases and like once, we learn how to do it manually. We try to apply some automated ways and that's where is the time to introduce some ai and ml stuff and makes the better use of that before that?

D

uh I would never recommend trying or jumping directly to ml without understanding how the data work and how the people work with the within the particle problem domain, so that that's pretty much my experience with with the ml stuff.

B

So um I I think it's funny right. One like I don't think it's a catch phrase yet, but it really should be which is like um you know. 50 of any machine learning project is data cleaned up. um You know. Maybe it's 80, um you know. Is uh it's a a classic uh challenge? um One thing I want to do is take a break here for a second and we'll talk about our sweet, sweet internet points.

A

B

You about that yeah, uh but then I had one more question for you all when we get back hopefully, um so let me share the screen again um and I will explain what we're talking about as well. Oh, that is not what I meant to do at all um all right. So these are our sweet, sweet internet points. um So uh we like to give out points for oh shoot, that's the wrong code, um so I will uh also get the correct code.

A

Not use the one.

B

On the screen, um but actually hey chris, why don't you explain the sweet sweet internet? Please.

A

Well, I will happily do that while you get the codes, so our sweet, sweet internet points are essentially awesomeness and a point thing, um so you know there's ways to get points from anywhere from watching the episode and just submitting the code for points to maybe issuing a pull request or issue against the uh repository that was linked earlier in the show today, which I will drop another link to as soon as I'm done talking. um So you know, participation basically equals points and using these codes, inserting them into the form will get you.

A

Your sweet, sweet internet points which we hope I have corrected or.

B

I'm about to throw in the chat.

A

Yeah hit refresh um or something uh but the the points right now are bragging rights, but they will have an intrinsic value of some sort in the very near future. Extrinsic, sorry, jesus. I keep getting.

C

This dressed up.

A

um Extrinsic value yeah, even in all hacking caught that it was last week's code, wow yeah,.

B

A

So the idea is build up your points eventually they'll be worth real stuff or you can exchange your points for real stuff. I should say um so langdon thank you for dropping the codes in there. I will grab the repo. If you want to uh brush up anything, I said feel free.

B

No, I thought that was pretty good uh yeah. So there's a like an activity, markdown page on the uh episodes repo which gives you some ideas for how you can participate to earn, earn points. um You know, as you may have you know, as you probably can't tell, but uh so you know some of the leaders on this board um have not just submitted episode uh uh points. So there's other ways to do it. uh Please check those out and that way we kind of catch up to the leaders.

B

um Obviously, if it was just watching all the episodes uh that wouldn't necessarily be enough.

A

So because you'd only have 3 900 points right.

B

uh Well, there's escalators, oh there's escalators, so um so at periodic points which I can't remember off top my head. um But uh when you have a certain number of episodes, uh then you actually get a bonus set of points uh for having collected that many episodes um and so uh yeah.

B

So it was just a way to kind of make the doing the show a little more fun uh participating in the show a little bit more fun um and we really like normally red hat, is very good at producing uh kind of prizes for things. uh For some reason this particular show is, has been preferred.

A

B

So we've gone through like multiple vendors we've gone through various uh t's and cs and challenges like that and you know, but we think they're. We, we believe they're ironed out, we're not quite sure why we haven't gotten over that last little hump, um but you know we're we're so close. um As you can see, we haven't even gotten this swag uh right. So that's to tell you something.

A

That should tell you a lot because normally like they're johnny on the spot, with the doing the thing.

B

Yes, getting the swag yeah, um so uh yeah, so the last question I want to ask: you was um kind of the uh you know your your mo. Your examples so far have mostly been about um problems.

B

um Are you currently doing or planning to do things where it's more like improvement, so in other words like because I'm an average openshift admin, I know to set up things the way the docs tell me to right, but because I have this particular confluence of events, I'm running you know 32 web servers and six databases, and you know I'm running php three seven. I don't know um because of that combination of things. If I tweak this variable over here, I will get overall, better performance. I would say: are you looking at stuff like that?

B

Or are you trying to cover the bread and butter first in a sense.

C

Yeah, all all at once, uh as I mentioned in the beginning performance, for example, is, is one of the four four categories that that we pursue. You know in some cases that can be uh that can be performance problems in in other cases, that can be performance, optimizations uh we're no different than insights. 4L. That way.

B

C

One and that's that's real specific, but I remember one specifically where you know there was there was a recommendation still is and when you have a when you have a certain rail setup and you run oracle database on top of that. Well up it was these. These licenses are expensive as hell right, and you know in many cases you know the the admins run them in a non-optimal way, and thus they have to pay even even more for licenses because they need to have more hardware and whatnot.

C

So uh you know we give recommendations about optimizing. Their setup increase their performance, uh you know, so that's what that's what we do across the board, both on both on rail and on openshift uh I'll, say that you know coming up with these improvements, it's much more difficult, so you will not see as many of them as you see, uh as you see with the problems because, like I said you know one thing about insights recommendation, they need to be specific. They need to. They need to tell you like.

C

This is how you this is how this is what you do now. You know like pointing out like hey, you have a performance problem here like you know that doesn't help much right and that's that's. That's part of the reason why it's why it's more complicated, but uh you know.

B

C

Definitely on the table.

D

Yeah- and another thing I would like to like mention is about like the the collection of the data, and you know we don't want our users to feel as we are spying on them, which you know the more you get to the workload specific things. The the the more risk like this might be felt. You know uh so that we need to be even more cautious and and making sure that we collect just enough data that can bring value back to the users uh and especially going to the workload specific stuff.

D

You know there might be more and more specific things that would need to be collected before we would be able to do this kind of recommendations. So, like one thing, that probably is the next on the table uh when looking beyond the openshift core operators would be looking at the combination or like getting the health data from the oil and operator space, which you know, gives you some some more and basically giving more recommendations that would be specific to to some operator and especially when, combining with the health of the cluster itself.

D

This can lead to some of the recommendations that would be more ortho specific, but like right now you know the the first thing that people should do is keep their clusters updated. You know, that's the the best way or one of the best ways how to get the most out of your openshift clusters, I would say, is having the latest version, because you have all the fixes and improvements that got there over time.

D

So that's why you know we want to make sure that the upgrade pass and the health of the clusters is safe, so that people, you know it's even no-brainer- for them to be on the latest release, which eventually also have the impact on the workforce running on the on those. You know the first one of the first things people get recommended when uh talking about any issue that is oppressive is have you tried to upgrade to the latest version and many times you know it can be the the way that fixes the problem.

D

So you know it's part. Our our job is basically make sure that the upgrades are safe as safe as possible and another one is also letting people know that when they upgrade there are multiple improvements that might be actually specific to that particular deployment. So we have a couple of recommendations that tell you specifically like we see you have this problem and we know it's fixed in extra version we recommend upgrading so that that's the the main point right now, as I mentioned.

D

No, yes, the workloads are something that we would like to get to at some point, but uh it's it's much more complicated than we would just you know uh uh deciding that we will do this as the next thing. uh There are more things to be considered uh with that.

B

And that's not my problem. That's more complicated, like you know, just just get on that. That's that's what I need um yeah.

C

B

It's on the table, yeah yeah uh so related to. That too, is another thing that I'd be interested in right. Is that and we're starting to run out of time, but it's kind of um you know how? How interesting is it? If I, as a single you know, end user customer whatever you want to call it um have multiple clusters, all reporting into insights.

B

um Is there interesting stuff that happens there right? um Where you have you know, maybe you still need to get into workloads whatever, but is there um you know, can you start to make recommendations? Like oh you're running, you know eighty percent of your utilization on one cluster, you have four of them that are bored. You might want to go fix that. That's like a stupid example.

C

Yeah um so well, the the first thing that needs to that needs to be. There is like uh what you saw in my demo was a single cluster view, but uh what uh advisor faurell has- and we don't at the moment, is a recommendation view like hey here is a specific recommendation, and here is a the list of clusters that are affected. That's currently a work in progress um and yeah. That's that's! That's where you start um other than that.

C

I don't know, like anything, can happen like we have tons of ideas. um We have a very creative product manager, I'll I'll say that yeah yeah.

D

And and marty also mentioned that we are not the only team like working with this and and making sure that there is, you know the most out of uh the the remote health data uh leverage. So even though it might not be in in realm of of us like solving this particular problem. As you know, resource optimizations, if you like, uh there are other things we'd rather have. That would be probably better to to speak to in terms of the cluster-wide views and recommendations.

D

On that you know I I guess even people from the advanced cluster manager uh would have you know things to say and and features on there or not uh on that particular uh particular problem. So we, as I mentioned we try to be. You know the catalyst of of leveraging the data for the benefit of the users, but it doesn't mean that our team would be you know, providing all the features that the customers could be benefiting from.

B

By way of explanation, the product manager is radically yeah, so he uh their product manager used to be my boss. So that's why yeah right exactly um so uh yeah uh anything else we should cover quickly. Chris. Do we have any uh impending questions aside from is pearl still in active use, which.

A

It's still very much. Definitely, yes, yeah! um No other questions. Really uh uh you know, except for that aiml one, which we kind of handled um right by.

C

The way what you saw even presenting on on the spike detection, just to give you an idea how fresh that is. That's being worked on right now, like a lot of team members who work on that very algorithm today. So that's fresh out of the oven.

B

Yeah as our as we often disclaim on the show uh this you know we we often uh are showing things that may or may not be from the future. um So uh you know, because we're we're trying to kind of show you exactly what we're you know working on what we're thinking about. You know that kind of stuff. So please, you know, take anything you see with a grain of salt. These are the ideas we're going down because they mean we'll actually be able to ship it, for whatever reason.

A

I would be remiss if I didn't say please like subscribe and share wherever you're following.

B

Oh yes, you're watching us from.

A

I should say um so: we need.

B

To use that catchphrase one of these days, I.

A

Know we need to come up with something incorporate.

B

Coffee, somehow, I'm not sure.

A

Yeah, uh I need an infusion of coffee.

B

A

As to why hot is empty, um so yeah.

B

Thanks so much.

A

Y'all, uh we really appreciate people.

B

Coming by um those are great talking about what they're doing uh yeah and you know I think this is this- is kind of next level. After what we've been talking about right, it's like you, get you get using containers, then you realize you need them orchestrated, and then you start to realize you know what can I do to optimize that environment that they're kind of running and that's where insights takes place?

B

And I think personally, I think that whole, like cloud.redhat.com kind of aggregated experience or whatever is getting really cool, uh there's lots and lots of stuff there. That is really useful. So if you haven't checked it out uh for any reason, you should go. Look at that in general,.

C

Thank you for inviting us linkedin. It was fun cool, awesome.

D

Thank you very much.

A

All right so coming up later today on the channel, we have the one and only andrew sullivan, with ask an open shift. Admin coming up here at 11, eastern 1500 utc, we'll be talking about authentication and authorization so tune in, for that I'm excited for it believe it or not just block everyone.

A

B

One has any permissions.

A

Only you needs access to this network.

B

As a former developer, give me admin.

A

Yeah yeah, so as big and fork mentioned stay cool out there. Folks I.

B

A

And here in the northern hemisphere it's been just absolutely brutal in some locations, so yeah stay safe out there, folks and we'll catch you next.

A

A