Red Hat OpenShift Ask an OpenShift Admin | Red Hat Livestreaming, 30 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ask an OpenShift Admin (Ep 63) | Recover control plane nodes and deploy custom Grafana

Description

Losing a control plane node is no fun and can be a shock for OpenShift administrators who then need to figure out next steps to recover the cluster. Fortunately, with OpenShift, there is a way to recover your control plane and minimize degraded cluster operations!

Did you know that Red Hat has plans to remove Grafana as a dashboard for OpenShift metrics? So at some time in the future Grafana will be removed by default, but you can still deploy your own instance and configure it to use the system Prometheus as a data source.

During this stream, we’ll demo how to recover from a lost control plane node and work with custom Grafana dashboards for the node exporter and more!

Learn more:
https://www.redhat.com/en/blog/custom-grafana-dashboards-red-hat-openshift-container-platform-4?sc_cid=7013a000002qG1LAAU

Try Red Hat OpenShift: https://red.ht/3uVxkya

Follow us:
Andrew Sullivan https://twitter.com/practicalandrew
Jonathan Rickard https://twitter.com/jrocktx1
OpenShift https://twitter.com/openshift

Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1

About Ask an OpenShift Admin:
Do you have questions? We want to answer them! Every Wednesday at 11am ET, hosts Andrew Sullivan and Jonathan Rickard sit down to answer your questions about Red Hat OpenShift.

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3rYz5ZS
OpenShift on GitHub https://github.com/openshift/

#RedHat #OpenShift #Kubernetes #ContainerPlatform

A

A

Let's get into the big topic of open source, something that we.

B

Actually have this is.

C

Hello and welcome to this week's ask an open shift. Admin office hours live stream. I am andrew sullivan, your host uh technical marketing manager with the open shift uh business unit and I am joined by my co-host, mr johnny ricard. How are you, sir? I'm good man? How are you?

C

uh I can't complain, it's pollen season here in north carolina, so if I devolve into a coughing fit uh my apologies. Was it last week where I like gagged on the stream? So hopefully that doesn't happen again that wasn't too good but uh yeah it. um I don't have allergies but the the pine pollen, especially here in north carolina it it's so prevalent that it creates this yellow haze, and even if you don't have allergies, it just gets like it. It's bothersome right! So oh yeah, yeah, apologies in advance!

C

If um uh to to, if I happen to you, know, start coughing or anything like that. I'll just go on mute! Try my best to go on mute before I. I devolve into hacking.

A

Yeah when I, when I used to live in virginia, uh my virginia beach area, the uh depaul, and you could see it like sitting on top of the vehicles and everything like I remember that it was. It was pretty awful.

C

Yeah yeah karate chop, it does, it looks like uh snow and it piles up, like you, know, snow blows in like on the porch and stuff like that. You'll like end up with you know, for lack of a better term, a a pollen bank, it's it's so thick and so prevalent so yeah. I I sympathize for anybody and everybody who's been around. um You know. For years I lived in augusta georgia, so I guess now it's um the masters is coming up right. It's the first full week in april, so oh yeah.

B

C

The indicator that it's it's masters time right, pollen polling time comes that's right.

A

Hopefully, hopefully, tiger makes an appearance like a like a real one. That'd be awesome.

C

uh You know it's funny, I I lived there. I I even went to a practice round once, but I'm I I have played golf a few times, I'm terrible at it, and I just I don't keep up with it too much. I know tiger was in a terrible accident and all but.

A

Yeah yeah I've been playing forever, I mean since I was like 15 years old and then um yeah he's. You know as he's coming up in the early 2000s. You know it's like.

C

A

Let's talk about it, yeah so he's pretty awesome. Yeah.

C

Well, he's he's uh he's right around our age right, so yeah early 40s, something like that yeah anyways. uh So this week um this week's topic is and you'll notice. We don't have a guest with us because it's just johnny- and I it's just us- we're uh we're going to be talking about a couple of pretty technical things.

C

So one is recovering from a failed control, plane node. So this is a question that comes up pretty frequently right, which is you know, hey I lost a control, plane node or what happens when I lose a control, plane node? How do I recover from it and the documentation for this is deceptively easy right it like you're kind of reading through it- and it's like? Oh, I just have to do these couple of things and nobody kind of believes it.

C

um So what I wanted to do is I I provisioned I created a cluster using the assisted installer and I'm going to deliberately you know, destroy one of my control plane nodes and we're going to see what it's like to recover from that.

C

So the other topic that we wanted to talk about is creating custom grafana dashboards.

C

So for a while now you may have noticed that we've been moving away from grafana for the metrics reporting inside of openshift, so particularly as administrators right. We we started off with the grafana dashboard back in you know when 4.1 was released and starting in, I think it was four six or four seven. You started to see the metrics dashboards be integrated directly into the admin console.

C

So there's a lot of the same information that was in grafana, just not in grafana, right and directly in the dashboard and the reason for that um largely has to do with permissions. So it turns out that the permission scheme inside of grafana there's no easy or good way to integrate it with what's happening in openshift. So if johnny is just a regular old user on my cluster, I can't give him access to just the metrics for the namespaces that he has access to right. It's it's all or nothing.

C

So that's one of the big drivers that were that we've been or behind that move away from grafana, but that doesn't mean that grafana and really all of that information, that's in prometheus, isn't valuable.

C

So I'm I'm going to show how to deploy grafana using the grafana operator and then we'll look at how to integrate that with the cluster prometheus as well as thanos and how to create some dashboards inside of there. So hopefully um you know we won't go. We won't have like a four hour stream or anything. Hopefully I can. If everything goes according to plan, I think we'll be done in about an hour hour and 15 minutes, something like that. Of course, for our audience, you can always ask questions at any point in time about anything.

C

That's on your mind, so whether it's related to today's topic or not feel free to ask us those questions. I think that we have added another streaming platform, so hello to all of the folks at red hat tv who are joining us in addition to youtube. I think we've got two youtube channels: red hat and openshift as well as twitch. So, whatever platform you happen to be on feel free to use the chat on that platform, the platform we use restream we'll make sure that all of those chat messages go everywhere.

C

So we'll be able to see those we'll answer. Your questions do our best to uh to help you out and the worst thing that happens. Is we don't know the answer? um You know I.

C

I can't speak for johnny, but for me I don't know a lot of things and I'm not afraid to admit that and uh worst thing that happens is we have to go back to the smart folks in engineering product management, uh our peers on our team stuff like that and uh uh get answers, and we can follow up in next week's stream. um We can follow up in the blog post, follow up on social media et cetera. So all right, I'm I'm through with my rambling. uh Let's talk about some top of mind topics johnny.

C

Yes, let me bring up my uh my notes here. uh So, let's, uh let's share my screen here. We've got three things to go through, let's see uh so the first one. Some of you may have noticed that upgrades between 4.8 and 4.9 are blocked. I actually saw somebody on the uh the kubernetes slack saying that oh I started this update process and I looked at the upgrade graph and it said to go from 4.7 to you know eight to nine.

C

I follow this path and I got to 4.8 and now it's there's nothing there and just so happened that, like we happened to block it, as they were doing that as they were going from seven to eight, so they couldn't get from eight to nine, uh so I'll post the link to this particular um or stephanie.

C

If you can post the link to this particular um kcs, so pretty severe bug, if you will, um but it has a low probability of being hit so out of an abundance of caution effectively, they have blocked up upgrades between 4.8 and 4.9, um and I think it says that somewhere inside of here, um basically to ensure that you know nobody hits this as they're going through.

C

So please, if you are on 4.8, you are considering going to 4.9, uh even if you're on 4.9 and considering you're going to 4.10 and the fast channel, remember 4.9 to 10 upgrades are still in fast, not unstable. Please pay attention to this kcs because it does have potential to impact you um so yeah. That's that's really I'll say about that. um I would assume I I didn't look to see. I would assume that there's uh an errata out about blocking updates.

C

I just saw the kcs and decided to pull that up as being our our thing here.

C

uh Our hope nine I'd like to ask if there's a convenient tool we could use to set up controlled traffic between nodes in a cluster need to debug some nsx latency issues. um That's a good question. I don't know the answer to that um johnny. Do you know of anything I'm trying to think like? Because I think what you're are you saying you want like a proxy, so you can monitor the traffic between the nodes or.

A

Yeah, I'm not really sure. If any tools I mean you could try and get into like the ovs tool and and try and like um you know like use like uh ipnet, you know the ip net commands within core os to actually like monitor the traffic coming across the bridge or the ovs bridge and see if that helps, but as far as like controlling traffic to certain nodes, so you can watch it. I don't know about that.

C

Yeah, that's um that's an interesting one. I'll have to think on that one I'll I'll ask a couple of the estimates that I know on on the back end. If we get a chance and uh our hope, nine feel free to reach out to me, uh andrew.sullivan at reddit.com and we'll see if we can follow up and get some information from, you generate enough traffic. So our vmware's just set up an iperf between two pods on different nodes and let it run indefinitely.

C

They'll love that.

C

Anyways um so yeah, please don't hesitate to reach out our hope. Nine. If we didn't get your question answered, um let's see so yes, potential scd data consistency issue um in 4.9, 4.10 upgrades from here 4.8 to 4.9 are blocked. So please be aware of that. Keep an eye on the errata. It will let you know uh both that it has been blocked as well as when it is unblocked all right. uh What's our next top of mind, the other two are um the other two are kind of minor.

C

I think um so one the the next open shift roadmap session. What we call the what's next session will be happening on april 14th, uh so that will be at, I believe, 10 a.m. Eastern time johnny- and I will be here along with a whole bunch of other- you know, live stream hosts and all that so feel free to come by. You know, watch the road map presentation, ask questions just chat with us.

C

We don't mind at all, but we'll be here for that, since it is on a thursday, we will be broadcasting our stream on the previous or the wednesday before that the 13th. um So is there any stigma associated with wednesday, the 13th? I don't think so. No, I don't think I think we're good yeah yeah. They moved this session on us. We, it was originally going to be on wednesday, so we didn't have a topic planned, so we'll have to figure out what we're going to do on the 13th.

A

We're clever, we'll figure something out.

C

Yeah, we'll figure something out. um Let's see, happy belated birthday to red hat um johnny- was that you that added that one in there.

A

Yeah so so I think it's march, 26th was red, hat's birthday and I I usually like have a ticker or something that shows some of these things and I caught it over the weekend. I'm like, I can't believe we forgot this last week so yeah last week was red, hat's birthday, uh we're old, um but yeah so march 26th, I think 1993 is uh when it started so uh happy belated birthday, red hat.

C

Yeah I, uh when I joined red hat was when was the 25th celebration, so nice time flies when you're having fun right. Indeed, all right, so uh I I don't want to waste too much time, because these are too uh I I in my tweet. I called them mini topics right they um from an administrator perspective, especially recovering from a control play node. It's not really a mini event right. We all kind of uh freak out a little bit right. It's it's! It's a big thing! It's a big deal when you lose.

C

You know one-third of your control plane and you know now you're down to no resiliency.

C

uh You know, as a old-school storage admin right raid, five right, that's that you lose one disk and it's time to uh start paying attention right. You sit up in your chair again and uh know that you need to do some work so.

A

That's right that stiffens up posture's all great you're focused yeah. We get it.

C

So, let's, uh let's, let's dig in let's start taking a look at what we got going on here. um So I I made a mistake and when I provisioned my cluster using assisted installer, I provisioned it using my employee account, which is not the account that I used to log into this stream. So I can't actually bring up the console to show you my the the proof that it was installed with assisted installer, but I promise I installed this using assisted installer, um so it is a uh it was deployed in the normal way right.

C

I have five nodes. Actually, what am I doing so I I to keep track of my clusters here I was showing johnny beforehand. I created a uh a little banner here, so here's my assisted installer deployed cluster and uh over here is my aws cluster that I'll be using for grafana in a minute.

C

uh So it's simple, five node cluster. um If we come over here to compute nodes, you can see. I've got my two compute nodes here. I've got my control plane nodes down here. um You know ultimately really straightforward, um because this is assisted installer, uh I used vsphere. So if we click over here to my vcenter nope, don't log me out. That was just in the nick of time. You can see.

C

I've got my 3v center or excuse me, five v center virtual machines so remember, assisted installer is basically a non-integrated or an on-prem uh bare metal deployment. So I did do things like I created dhcp reservations for the ip addresses, as well as for the host names and all that other stuff. So all of the things right, there's, uh oh and the um uh the vips right. So the ingress and api vips, so yeah um everything is in place. All I had to do is click through the assistant installer.

C

Let it go and deploy and do its thing and literally the only thing I have done in this cluster is put in place. This banner I haven't touched anything else, including you can see. I haven't applied the cluster upgrade.

C

So, let's find out what happens when we break it right. So the first thing I want to look at here, real quick, is so the documentation so from a documentation perspective, I'm just on the 4.10 documentation.

C

I have browsed down here to backup and restore and then control, plane, backup and restore, and we have this replacing an unhealthy fcd member, and one thing that I want to look at here is.

C

Right now we see that we have three members available right. My etsy d cluster is healthy right, so let's go and break something.

C

So I'm going to actions, and I want to power and power off not even going to give it a graceful shutdown.

A

So, are you doing this from the perspective that you don't have any std backups you're just going to go straight, yep.

C

So, okay, so I haven't created any ncd backups or anything like that, because you don't actually need them to recover right. So scd backups are helpful in a couple of different places or reasons right. So let's say that I lose two control plane nodes. Let's say that I.

C

I'm trying to think let's say that something goes horribly wrong and I need to roll back the entire cluster to a point in time. Right, ncd backups are useful for basically seeding scd with the data. That's in that backup, which is created at a point in time.

C

In this instance, I still have two of my control plane nodes and if I come back over here and redo, this command it'll take it a couple of minutes for kubernetes to you, know, remember or figure out that it's dead, but I can do a uh so see, get node. I can force it um so there. It says it's not ready all right.

C

um So essentially it's with two control plane nodes still in place. Everything is still functional. Scd is still read, write, it hasn't gone read only because it has lost quorum or anything like that. So the behavior is effectively when this node comes back. When this node is replaced.

C

The other two nodes will populate its etcd with the data that it needs. So I don't have to recover that from a backup. Now I can, and depending on your cluster, especially your cluster size and your cluster age and by size I mean not just number of nodes but number of pods number of things. You know objects in etcd. That may be a good idea. If my scd database is you know tens of gigabytes in size.

C

That's a lot of I o that has to come from these other two control plane nodes as as well as being written to this control, plane node that can put a strain on the back end infrastructure that can lead to increased latency et cetera, so it may be worthwhile for me to you, know, seed or stage some of that data, but it's not required.

C

So just keep that in mind. um So, let's see if this is yep so two of three right, so it finally realized that hey one of my etcd nodes is unhealthy.

C

We can see here that it was marked as not ready. So let's follow the documentation, so you saw I just copied this. This command right check the status of the of the scd members available. We see that only two of the three of them are available. Just like we see here. So let's determine the state of the unhealthy member, so we know that the machine is not running. I don't need to check this.

C

So one thing to note: the documentation here is very much in line with or very much assumes that you're using ipi and you have the ability to basically provision virtual machines or nodes right machines in your infrastructure on demand. So we're not doing that. We, you know we did a bare metal install using assisted installer, so I I can kind of ignore these commands about machines right. Yes, our node is definitely not ready.

C

So, let's check the status of that.

C

See it's declared our control node zero, zero as being not ready, and we can still see over here in node. It's still listed in not ready status, so we'll scroll down a little bit. um We know that std is not crash, looping, it's just not there.

C

So how do we replace this now unhealthy member? So replacing a member whose machine is not running or not ready? This is the status that we're in so let's scroll down and follow the documentation, so we're just going to copy the command here.

C

Paste it in, but what's interesting to me here, is that you notice that this still says it's running. I think this is because it has been less than five minutes since I took that note down, so it doesn't register that the workload isn't there and it won't try and reschedule it until after that expires.

C

I could probably force it by going ahead and doing an oc delete node on this node.

C

See what happens there so yeah so now and now it's registering that those are terminating they're, not actually terminating, because again the node isn't even there right. It's not powered on there's no way you can talk to it, there's no way that those were running before, but at least we can see that hey.

C

There is something happening there if we come down here, so we're just keep following down, so I'm going to copy this part of the command, and what I want to do here is: I want to connect to one of the two surviving sed pods right, so this is going to be our, so we know that these are the two good ones so we'll connect to this guy oops.

C

All right so now I'm inside of the scd pod. We can do this member list command. We see our three members here, so here's the one that we took out this is the bad one.

C

So we can see the example output follows the same thing and now we need to remove one of our members. The fail the failed xcd cluster member so remember, remove so control. Zero. Zero is the one that I failed, so we'll go with the id from that go ahead and remove it and just to double check. We'll look at our member list again and we have just the two remaining valid up: control, plane, nodes um johnny by the way feel free to ask interrupt me with any questions that pop up so.

A

Yeah I'll just reach out to the group to let them know if they have any questions to ask them and I'll make sure I get them out.

C

I am not paying attention to chat at all, so yep.

A

C

All right um so remove the old secrets right, so I removed so. At this point our node has failed. I have removed the failed node from the fcd cluster. So now I need to go into uh and remove our secrets so rep for control dash zero, zero.

C

So I suspect, and the reason why we're getting a delay here is because this is the api vip bouncing around a little bit. We'll probably take it a couple of seconds.

C

And I don't know why it does this. It did this to me yesterday when I was testing this as well, where it kind of goes up and down a little bit as I modify things.

A

It's probably the response in the load balancer, where it's trying to like cycle through yeah.

C

So I suspect that, had I done a quote-unquote full upi deployment or non-integrated deployment, where I have an external load balancer, um you know responsible for the api and all that um that would have been fine. We wouldn't be seeing this. This is simply because I'm using the vip the keepel id managed vip for these ip addresses.

C

You know I probably should have checked, which one was hosting it just.

C

Whoops just to avoid the drama right.

A

So one thing that's pretty cool about this too. Is that like, if you were to follow um like the kubernetes documentation and do something like this with kubernetes as well like the process is very similar if you're going to do like upstream kubernetes etcd, you know restoral stuff, like that backups.

B

A

C

Well now I have no idea what's going on, what's what's happening here uh so.

A

That one's karate uh karate chop asks if a node was to fill like a motherboard, for example. um How long would it take openshift to realize that it's failed and reschedule the workload um that was assigned to the node to the other nodes? I think it's five minutes right like where it goes through it. It's like yeah.

C

Yeah, so by default it's five minutes. It'll declare the node unreachable and then it'll reschedule the workload after that, the exception being with pvcs.

C

So the mounted pvcs there's like a whole, another timeout that has to happen there before it will mount those. um So this is usually you know the the openshift fert guys make a big deal out of that, because um you know with a virtual machine. I want to be able to reschedule it as soon as possible.

C

So how do I reduce that aj time for a virtual machine? And that's where the whole poison pill operator thing came from of my node went down? I don't want to wait five minutes for kubernetes to say it's not reachable, and then I need to wait. You know, somewhere between one and five additional minutes for the pvc to free up.

C

Instead, I'm going to use poison pill operator that is going to you know simultaneously reboot the node and then remove it from the cluster. So that way, it immediately triggers a reschedule of all that workload and because the node is rebooted, we can we're confident that the pvcs aren't mounted. So it can immediately remount those as well.

C

So all right, so you see that my uh api momentarily came back. I don't know what's going on here, it's weird.

C

So yeah our hope, nine um node health check, uh so node health check in combination with um a machine uh remedy. What is it, what is that called um so there's an an action that it can take where it will with ipi.

C

Basically, when node health check fails, for you know, say three times in three minutes: just delete the node and if you're using like bare metal ipi, then effectively, it will reprovision the node and rejoin it to the cluster, um just because ipi expects it to be there what's going on here.

C

This is like uh this is I'm feeling, like I'm doing, one of your your.

A

I I wasn't gonna.

C

A

I wasn't gonna say anything I was like. uh I was just gonna, let it go.

C

I was just looking like um you know here: let's, uh let's refresh vsphere, so it the ip is on here like what's going on. Why? Why are we bouncing around and I literally just reprovisioned this cluster this morning? It should be all right. It's there for the moment, we'll see what happens all right. So let's try our oc get secrets again um here you go! Oh so you get secrets. So here's our three secrets for the control: zero, zero node. So I need to delete those secrets.

C

And open shift at cd.

C

And we'll just take care of all three of these in one command.

A

Yeah and andrew would never get a chance. um Thomas fall was asking. uh Hopefully I said that right um was asking if the timeout is customizable for, like that five minute. Sync.

C

Yes, but so technically, yes, certainly upstream, um I don't know if openshift supports modifying that and that's.

A

C

Thomas p, hall.

A

Oh thomas p, okay yeah. That makes sense so then thomas p, hall.

C

So thomas hall, yeah so yeah, I think it technically is customizable it's uh it might be a scheduler setting. It might be a kublet setting, I'm not sure which um so yeah all right. So our secrets are gone now. Switch back over here so delete the secrets. This is what I just did. They just did it in three separate commands in the docs, and now we have this very generic delete and recreate the control plane machine, and then they give you this uh this set of instructions for how to do it using ipi right.

C

So, oh so you get machines and here's how to delete the machine and then we're going to create a new one. So, basically, with ipi and machine api integration, you can manually create the machine, object and trigger it to create a machine. For example, they did this in aws we're not using aws right. We don't have that option, so you saw I deleted the node from the cluster right. I don't have a control, 0 0 any longer.

C

So now. What do I do? So? The first thing that I need to do is I need to extract. I need to get the machine config. Excuse me the ignition config for the control plane nodes. So if I do a history and grep for extract.

B

C

Extract correctly.

C

So I have this command oc extract from the machine api. Let me copy it out here so make it easier to read too oc extract from machine api namespace, a secret called master dash user dash data dash managed.

C

I want specifically the user data key, I'm going to extract it to the terminal so to dash, and then I'm going to redirect that into a specific file master.ign.

C

So let's go ahead and do that and it appears that my api is still flip-flopping. So it's going to take it a second to timeout.

C

Yep come on, you can do it. Buddy.

A

Come on little buddy.

C

I have the jeopardy theme song playing in my head right now. Indeed,.

A

Indeed, I could start singing, but I think I think that'd be bad.

C

I appreciate your singing johnny. Don't don't be shy.

A

Somebody does oh come on.

C

So I I suspect, what's happening behind the scenes here is the vip is in place, but for whatever reason the api pods are bouncing up and down. I'm sure I could troubleshoot that I'm just going to try and muscle through this.

C

So now you can see it's it's unhappy right now,.

C

It's definitely struggling to respond there. We go all right so now, if I do a quick uh cat on the file I just created and put that to jq, so we can read it. You can see it's just like if you created the ignition config for a control, plane node using openshift, install right, it's going to say, go here for your real machine, ignition config and here's the certificate authority that you need. So you don't freak out right, so very straightforward, very standard from that respect.

C

So all I need to do now is: I need to host this on a web server, because the way that we are going to recover this node is basically to provision a new node. Just like I were doing a bare metal, non-integrated installation right. So I'm going to host this on a web server. I'm going to take my virtual machine, I'm going to boot it to the live iso, and then I'm going to do a core os, install and point it to this uh config.

C

A

Real quick enter before before you move on. Can you um can you kind of go back to the console and show everybody like you know, just why you're being patient and then how you knew that things are ready? You know what I mean, because I think that this is a good chance to show like hey. We hit a wall, we had a problem but instead of like you, know, freaking out, and you know, killing everything and restarting like you, you went and looked at some things. Can you kind of walk through that really quick.

C

Yeah so and yeah so right here, you can see immediately the issue right.

C

So when we remove that control, plane node, it has now caused the operator responsible for kube api server and probably openshift api server too um yeah. You can see it's only 81 seconds right, so both of those are they're bouncing up and down because they want to recover back to their state. So I'm sure that I could do something like you know: oc get pods dash in cash.

C

I do a describe on my cluster operator. It'll, give me this related objects.

C

So if I look down through here, I have this namespaces and I'm looking for open shift, kube api server right so that one looks like the right one. So we'll do an oc get pods actually it'll be the operator. That's bouncing up and down so kube api server operator. Oops got my dash in and from here I can look at the logs, so oc logs.

C

And yeah you can see it's it for whatever reason, I'm sure I could dig through this and figure it out as to why it keeps wanting to do a revision. um You know transition to 12 why it keeps wanting to do this, but essentially you know it was a.

C

It was an assumption and a safe bet on my side that the api server was was freaking out and is trying to go through its thing in order to get to um the right revision or the right config so and when it does that it takes down pods. So, even though the vip isn't bouncing around, like I had originally said it's the pods that are the api server that are bouncing around and again I don't. There is a reason um why the api server pods take so long or why updating this takes so long.

C

I think it has to do with there's like a codependency between api server and authentication, so api server, bounces and then authentication bounces. An api server doesn't declare itself ready until authentication comes back, and then it it's something like that, so it just for whatever reason it takes these a long time to um you know longer than most others to bounce.

A

C

A

I just wanted to. It was a great opportunity. I thought to kind of show like sometimes you're going to have a problem when you're you're expecting a certain output. You know, and things are going to go wrong eventually, and you know to just understand how to how do I debug this quickly or how do I look and see it and get that warm fuzzy that, like okay, at least I understand the problem, yeah and karate chop? I I believe that there is a uh a cheat sheet I'll try and find it.

A

But if not, then uh I'm sure that we could probably slap one together for like troubleshooting and stuff like that, yep.

C

Yep uh there is uh there's a kcs. um We can dig that up like there's a I'll have to do it in one of my other browser windows and look it up. um So all I did here just scp that ignition file that we extracted a moment ago- and I pushed this over into my helper node web server.

C

So let's open a new browser window and we want to browse to this just to make sure that it's there that it's available all right, 503 unavailable. So, oh because I need in the right port and I need to put in the uh location, chin, orv dash master. We can see oh 330, so at 11 36, which is today, and if I click on that, I get my control plane ignition file json. So everything works there we're good to go all right. So basically we have everything we need staged, except for our core os iso.

C

So I'm going to go to mirror.openshift.com.

C

And it drops me into this mirror so really the way that you would want to do. This is to go. You know either through console.redhat.com or to the customer portal access.redhat.com I'm cheating, because I can um and just browsing directly to the mirror and I want to go to dependencies and then we want core os 4.10 and there's only one.

C

So all I've done- and I've already done this- I downloaded in my instance the live iso. So if we look through here, we have live.iso.

C

So when I boot the virtual machine to this iso, it will drop me into literally a live environment right just like any other live iso, and from here I can configure the nodes and do a bunch of other things inside of it. I could also do this by booting to the iso and using a kernel parameter but potato potato, so I've already downloaded that iso.

C

If we switch over to our data store view here, we can see that I've got my core os 4.10.3 iso.

C

So, let's come back over to our failed control play node rather than completely destroying this virtual machine and going through that process. All I'm going to do is remove our hard disk so delete from datastore and we'll let that do its thing.

C

And then we will just give it back a new disk, so I'm gonna give it 120 gigabyte drive. I want it to be thin provisioned because I don't have enough disk space and then down here for my cd dvd drive, I'm going to connect it to a data store iso file and I'm going to connect it to our core os iso and make sure that we tick this connected power on okay.

C

Give it a second to update its config, and then we will power it on.

C

C

C

On I miss the button here. What happened.

C

Attempted operation cannot be performed in the no. I don't want to send details.

C

What happened? uh What am I looking for here? I want tas.

C

Why did you not turn on.

C

One more time no okay, come on vmware.

C

You've got your disk. You've got your iso file connect the power on okay.

C

Maybe I will have to oh, it already is powered on, of course, web interface.

B

C

You know I'm when was when did we first start using the vsphere like way back when it was the desktop thing, and you didn't have to worry about it, refreshing right and now? Oh.

A

Yeah yeah, I would. I would just think that same thing like I remember, the old ui.

C

So yeah over here it shows not powered on, but it is powered on anyways. So all that happened while we were while I was banging my head against that- was it booted to the live iso, and you see it drops us directly into this. So one thing to note here remember that I used a dhcp reservation, so it already got its ip address and its host name.

C

So if I had completely destroyed this virtual machine, all I would want to do is go in and create a new. You know change the old dhcp reservation to use this one's mac address.

C

So I want to do exactly as the command here says: coreos installer, so sudo core os installer install to do ignition.

C

Url, I can't talk and type at the same time,.

A

So andrew g kumar asked if we could do an episode on um the openshift update service operator in disconnected environments.

C

um Yeah we we've we. We came close to doing that when we did the deep dive right and then we ran out of time um so yeah we can do that. uh Where was that url nope? Not you.

C

All right, my I'm having.

C

A moment uh so ignition or v-master.ign uh nope here we go ignition, slash o r v, dash master dot. Ign- and importantly, I want to do because this is both http url as well as I'm not providing a sha hash of the file. I want to do dash dash insecure dash ignition.

C

So where did I get that from so? Let's come to our documentation here I want to go to install and I want to go to installing on any platform and I'm going to open that just in a new tab. So I can keep our hcd recovery docs here and if I scroll way on down here to installing our archos and if I scroll down through here right installing using an iso image.

C

This is what we're doing so see how it describes this sha 512 sum blah blah blah, and then you can take that and as we get down through here, it'll say: run coreos installer install, provide the url and then provide ignition hash. So I'm going to take the risk and say that nobody has hacked my ignition. Nobody is modifying it.

C

So I'm going to trust it to not be secure and the way that we work around that if we keep on scrolling down apologies for the scrolling, if we scroll all the way down through here, we'll eventually hit the part of the docs that has all of the parameters for coreos installer. Here we go so coreos, installer, installs, sub command options, and I have this down here. Insecure ignition, you can say, allow ignition url without https or hash, which is precisely what I'm doing right so dash dash insecure ignition.

C

Once I find the right window and we hit go oh right and I got to provide a location where it's going to install to um yep. So there we go so.

A

B

A

9 has had enough of the andrew and johnny stream and he's wondering when we're gonna bring ort one back to do some more uh to go through some more slides.

C

uh You know, maybe we should do that on the 13th. Maybe we should see if uh not.

A

A bad idea, yeah available.

C

A

Did it while it's still fresh yeah.

C

So who? uh What are we talking about next week next week next week? Is uh security with um kirsten newcomber? So if you're not subscribed, go ahead and click the button, so that way you know when that comes up. You can also follow us on social media. John and I've been trying to do a much better job around posting topics for the upcoming streams on twitter and linkedin and reddit, and all the places so and for red hatters. I think um so we have a tool called bamboo.

C

I think we've also been posting those into bamboo. So you can it's another way. You can stay uh stay informed of what we're doing so cross installer ran here. I don't know what it's doing in the background, but it takes it a few extra seconds to go ahead and finish. Writing the core os install to disk once that's done. What I'm going to do is shut down this virtual machine and disconnect that installation, iso and then power on the vm, so the next steps after that are going to be. Basically, we wait.

C

um So as soon as I hit I power this off and and restart it, I'm going to move over to the grafana side of things, because, basically we wait for the csrs to appear once we approve the csrs, then basically magic happens and magic in this case means operators so effectively once it sees the new control plane. Node join the lcd operators and everything else will automatically redeploy everything to it and it'll just work right. We wait a few minutes and and uh it'll recover, it'll it'll heal itself back right.

C

There we go so writing it, and it's back it's done so I'm just going to do a shutdown h now, so we can power off our virtual machine and that's done so we'll do a refresh here.

C

Let's hide this do a refresh here, so there it's powered off so edit settings. We will change this back to client device.

C

And power it back on.

A

So our hope 9 said that he likes today's topic. uh It's the moment that most people dread the most. When you know things are supposed to happen and then they don't or you're waiting for something to finish, and you have no idea what's actually happening in the background.

C

Yeah, all right so vsphere is doing its thing. It should be powering on here. Come on come on console anyways, it's a standard, boot you'll see it go through its process. I'll leave that open for now, um but all we're waiting on at this point is uh we: do our oc.csr csr we're just waiting for the csrs for this guy to to appear in there. It'll take a couple of minutes, um so I'm going to change topics over to grafana and I'll check back in a couple minutes. Johnny!

C

Don't let me forget so I'll check back in a couple of minutes, we'll approve those csrs and it'll take another five or ten minutes for everything to heal itself, and then we can see and uh just to check real quick here. So one you see, there's all kinds of errors happening inside of my console for this guy. I come down here to nodes. You can see I'm missing a node right, we'll see this pop back in once it gets there.

C

So one thing um if you're, if you're sharp eyed, so I I installed this cluster through assisted installer and it is a it's on vsphere right. You all just literally saw me do this on vsphere.

C

I don't know why- and I just discovered this yesterday but like the nodes are showing up as under a machine set and they show up as bare metal hosts but unmanaged. So I'm I'm not sure what this means yet. But it's interesting to me um so maybe johnny another potential topic for the near future is we should get the assisted and seller folks on um and and have them walk through a few things and talk to us about. You know kind of what exactly what we're seeing here but yeah.

C

If you saw there, you know my nodes, some of them have this machine associated with it, and I I don't like I said I don't quite know what that means.

A

Yeah that'd be awesome to have them come in and clear that up.

C

Yeah, so, okay, all right we're just gonna sit back and wait a few minutes for that to happen, and meanwhile I'm gonna switch over to our aws instance, and I just used aws for this, mostly because my lab isn't big enough to deploy two full clusters. So if we look down here same thing as before, I've got five nodes, actually six nodes, because it's aws, so one two three control plane nodes, one two three compute nodes inside of my cluster, um so this one is pretty straightforward right.

C

So if I look at my observe- and I go to metrics here um so this is where I can query prometheus directly for for information right. So I can do like node um there's all kinds of stuff in here, so I'm just gonna select one of these at random run my query and you can see it spits back all the information. That's in prometheus.

C

I can go to dashboards and I have this. You know you know we'll switch to the ncd1, so I have all of my sed dashboards spread all the stuff that I've come to expect and know, and if you know or if you're aware of it, we do still have grafana installed right. So if I go to networking and routes and then if I look down through here, I've got this grafana right and I can click on this guy and go through and accept all this stuff um yep log in as me.

C

Maybe yes, I do want to assign myself those permissions. Thank you for asking um and I do yeah. So I can go in here and write it's. It's all the same dashboards that I come to know and love and expect, but one this instance of grafana is read only and two. This is what we have deprecated and basically, we've declared is eventually going to to go away right. So what if I want to you know, modify this.

C

What if I want, for example, I I want the node uh dashboard because we're running the node exporter, so there's a great default or community created, I guess dashboard for the node exporter. That shows all kinds of information about my nodes.

C

What if I want to see that so this is what we're going to address is how do I deploy my own grafana so that I can customize it so I can get whatever information I want out of my particular cluster, so step one is to go to operator hub and for grafana you'll notice that this is a community operator. So it's gonna remind me that it's a community operator.

C

So, yes, we're going to go ahead and hit install, so I'm going to create a project for this, and my project name is very creatively going to be named. Grafana.

C

And that's all I need to do to install the operator right, so we'll hit install and that'll take a couple of minutes. While that's thinking about it, we'll come back here and check on this guy. Still not there. Yet you're still doing your thing yeah, so you notice it hasn't, picked up its host name yet and stuff like that. So in fact it got a completely different ip. That's strange!

C

I'm just going to trust that to do its thing and we'll we'll see what happens.

A

So mike murphy was asking uh this is about the canary route operator um asking if the route checks are performed by the stat port or um http or https. I think it's http that it's doing the stat check on or that's doing the check on, but I could be wrong.

C

um Yeah, that's a good question. I don't know um how would we check that we can check that by going to.

C

There is a crd for that and I'm trying.

A

To remember what.

C

A

The operator is called canary operator or something like that.

C

Think it's probe under monitoring no yeah, I'm trying to remember there's um all of those should be defined somewhere in here uh here we go and you know under ingress, congrats prometheus, no insights.

C

Now that's for the dashboard, so we don't want a prometheus rules object. What do we want? We want.

C

uh We'll have to look that up. It is possible to check I'm just not remembering it off the top of my head.

C

A

C

Just regularly.

A

Remember like if you had like a redirect with http to https, it would fail back in the day so.

B

A

We'll look it up, though, mike murphy we'll get back to you.

C

All right, so my grafana operator finished deploying you see here, I'm in my grafana project. If I click into this guy you'll notice that there are four crds that it is responsible for so grafana, which is literally an installation or a deployment of grafana.

C

We have a grafana data source, we have a notification channel and a dashboard, and I just now noticed the inconsistency. That's going to make me crazy. If there's a space here, but not here or here.

C

So data source, as you would expect, is literally a data source, a data source right, hey prometh or grafana. Where do you want to get your data from and our instance is going to be? Prometheus dashboard is hey. What dashboards do you want to make available? So these two crds? Actually, these three crds are all utilized by the grafana instance to basically precede or get data for it. So this can be useful for a couple of different reasons, so, one by default.

C

When I create an instance of a graphana, it's not persistent, there's no pvc associated with it or anything like that. Instead it deploys it and then it uses these crds to populate information in it right. It'll, look at the data source and they'll, say: okay, I need to go and grab. You know. I need to configure this inside of my deployed refund instance same thing for dashboards, etc.

C

So, let's look at a couple of things. So, first, let's go to our grafanas and I'm going to create a grafana and I'm going to use yaml view because it's way easier for what we want to do here.

C

So off to the side here- and I cannot make this any bigger, can I I don't think I can so apologies that it might be a little hard to read but we'll walk through what this looks like so first, I'm going to give it a name and then I'm going to provide a bunch of different information here.

C

So how did I determine everything that I want from here so one I can use the form view right and in the form view I can go through and I can fill out each one of these things and it'll do kind of exactly what I want right here. Admin password is admin which you can see over here. I set admin password equals admin.

C

Alternatively, you can go to github for the grafana operator, so I will paste that into twitch as soon as I find the window we'll paste that into twitch. So inside of here I can look at all the documentation.

C

That's in here I'm not going to scroll through that because I know where I'm going and what I want to do is go to documentation and I want to go to in my instance right grafana.yaml, and this gives me an example- and I think this one I almost exactly copied out um so one thing I haven't found- and this is maybe andrew just hasn't been looking close enough- um I haven't found a good like what are all of the fields.

C

What's the description for all of the fields available in that crd, so I did find so where is that file at?

C

So if I go to fauna operator and I go to api and I want the integrate lee v1 alpha one and I go to grafana types, I think that's the file. I want yep grafana types, so this is basically the the go level descriptor for the api, so we can see, for example, if I search for security.

C

And hold down like here's, my grafana config security, api structure and down through here you know: here's our admin, user, here's our admin password! You know all of the stuff that I need in order to do that. Like I said I haven't found in the documentation here on github like where all of this is described at so this is just I've. Sadly, looked at enough of these uh uh crd definitions, api definitions that I had a good idea of where to look- um and I I can see in the examples um so just for reference.

C

Let's go all the way back up here to the top graphene operator.

A

And wally was asking why you couldn't get this information out of the um like explain. I I think you could get like the top level out of the explain, but like the actual definition and descriptions of what they do, is what you're actually looking for right.

C

Into an oc, explain: grafana dot spec I can talk and type so spec dots.

C

So what I want config so yeah, that's a great point. While um you know you can see here all of the objects all the stuff that's inside of here and I can do like you know, auth.

C

And security and dot admin underscore passwords, um but you can see again it's it's super basic what's in here, so this is um excellent. Point. Walid of this is the you know integrated. You know open shift cli and we can do the same thing where in my window go you can see over here um in the schema v details, and then I can go down here to config and then I can go down here to security.

C

Alphabet is hard, so here admin password right, um so yeah I can look in here. um That is an excellent point. I should have probably started with that. Instead of digging through go code but anyways, so I'm setting a default username and password telling it to log out to the console telling it to enable an anonymous login right. All this stuff is standard, grafana, config, disable the login form disable word, keep the login form, but disable the sign out menu.

C

So data storage for this one, I'm going to request a pvc, so you can see it's on rwo pvc coming from our gp2 csi storage class, which is configured by default with 4.10. It is not the default, but it is configured by default.

C

I want it to create an ingress, so we'll have a route for us and then it'll also create a uh a service for us. So let's click create and let's see what happens, and while that's doing its thing, we'll check on this guy again. Here's our first csr.

A

Before you before, you prove that real quick can you do like oc, get nodes.

C

So you can see with this first certificate. It doesn't appear at all because it's the node bootstrapper, so oc adm certificate approve and get the name.

C

And should happen here is uh it'll pop in. I think we'll pop in after that.

C

So here's our second one. You can see it's only four seconds old.

A

Yeah, so the reason why I was asking for that is because, like you know, sometimes we have people that will turn the clusters off over the weekend right or on a friday night or whatever, and so they'll come back on monday and they'll have their cluster. Essentially it's up, but it's really down, um and it comes down to these things where the csrs need to be approved, because they've timed up.

C

Yeah, so I I did do um so a couple of things here, so one I probably should not have approved this csr and the reason for that is because you see it didn't get the right hostname, and this goes back to what I was saying a moment ago. Right it never picked up or it picked up a whole new ip address and that doesn't have the uh dhcp lease associated with it.

C

So that could be something with my dhcp server and it doesn't want to give out the same lease to what it thinks is a different host. I I don't know what's going on there, but because that didn't get the right, node name, I'm not confident that this is going to recover, because it's going to want to try and look up the right, I'm going to it's going to want to try and look that up by hostname and that's going to fail. So um in theory, assuming everything was correct.

C

So let's say I I proved this one. I saw oh wait, that's not right! What I should have done is basically redone that whole process of reprovisioning the node and made sure that it got the right, hostname um yeah, write ip and write hostname um and at that point again, basically, if everything was working, if this was showing up as oc or orv control zero zero.

C

I don't have to do anything at this point. After a couple of minutes, it'll come back and everything will be fine seo. You can see. It does register that there is a new control, plane node, because up here you see it's it. The role is master, so it most of the operators are going to register that and they're all going through, and you know trying to reapply the configuration but again because this thinks its name is localhost and there is you know. Obviously localhost isn't going to work when it's trying to reach a remote node.

C

um I I'm expecting this to fail so we'll let it go while we finish up with our gofundme over here.

C

So, let's come down here to our workloads and we've got our grafana deployment up and running, and if I come down here to networking and routes, we've got our route and I can click on this guy and yes, I know I need to accept the certificate and I'm in grafana right just like that.

C

So let's go to our sign in and I'm going to sign in with admin and admin and because it's the first time I'm logging in it wants me to set a new password which I'm going to set, and you know here we here we are right, I can go in and I can configure all the stuff that I need.

C

So how do I configure this to access the internal prometheus or the internal thanos endpoints? So in order to do that, I'm going to need a service account.

C

uh So if I do oc project to go into the graphana project, get service account and inside of here I've got this grafana service account, you can see, it was just created, so there's a blog post that was created um a while ago by some of the folks.

C

Let's see, I want to copy this link and bring it up over here and I'll share this with me on the back end, so she can share it out so inside of here, this blog post describes kind of exactly the process that we're going through. So I want to add the cluster role. Cluster monitoring view to our grifon service account so we'll copy that command all right. So we see that we added the authorization and now I need to get a service token.

C

For excuse me, a bearer token, for my service account, so oc service accounts is the command that we're working with here. Git token for the grafana service account that we just looked at here and in the namespace fauna someday I'll, remember how to spell graffon on the right first time and it's gonna spit back at me this guy right. So this is what we use to authenticate over to or internally over to prometheus.

C

So I can come back over here and, I can say, add data source. I want to add a prometheus, we'll give it the lovely name prometheus.

C

So I want to I'm going to skip over this down here by default, because I want to come down here to headers and I want to add a authorization. I think author authorization. Now I don't remember, is it in here authorization.

C

Authorization and then my value is going to be error.

C

I know it's dotting it out so b e, a r e r space and then that big ol long string that I just posted, um so we see here and where I got that from- is this right.

C

So all I'm using is that now the reason why I'm doing it this way and not with this grafana datasource crd, is because I have a pvc which means that this will persist. So I can do it this way, if I so choose, and it won't affect anything.

C

If I didn't have that pvc there. So none of this was persistent then I would want to do it with our crd from over here and I'll show that in just a moment, but I wanted to do it the harder, but I think the way that a lot of people will prefer, because it gives them that you know kind of traditional full grafana experience.

C

I wanted to do that first, so the last thing that we need is the url to use. So if we look in here, this is going to point at the thanos querier right, but I can do a oc get service dash in openshift monitoring, and you see that I have this prometheus kubernetes right. So all I need for that is um the same. You know I follow the same url pattern of prometheus kubernetes.openshiftmonitoring.service.

C

So just basically replace kubernetes dash or a prometheus kubernetes in this line. While I was just looking over to the side there I cheated and grabbed that, so our url is going to be prometheus, kubernetes, dot, openshift, monitoring, dot, svc, port 1991 and now I can save and test, and why am I getting a bad gateway.

C

And let's see what's going on here, name prometheus url,.

C

That's what I get for not having specifically tested this one before.

A

Thanks for joining walid, hey andrew real quick, I know you're like trying to debug something um uh de love. It asked if, uh if there's a way to create custom dashboards for one of our uh for our application, do I need to have anything specific enabled on the application. Ie slash, metrics endpoint. um I think rick did a good job of answering this, though, and said that, like essentially, you just need to enable the metrics collection uh on that workspace.

C

Yeah, so what you would do is, um if you want to do it through the cluster monitoring stack, you would go in and you would enable user workload monitoring.

C

Right, enabling monitoring for user defined projects, so it's literally just a config map setting and what that'll do is that will deploy a second prometheus instance, and you can then use the standard crds in order to go through and configure metrics endpoints, and you know the rules and all that other stuff, and then you can either query that directly. um So, just like you saw me do a moment ago, there'll be another, I don't know what it's called: it's like prometheus user or something like that.

C

So there'll be a second prometheus or you can query the thanos in point. So the reason why I'm choosing to use prometheus here instead of thanos is because thanos slightly alters the schema. So if I want to use, for example, if I browse to uh what am I looking for on dashboards.

C

I go to here like let's say I wanted to use the node exporter.

C

C

So if I were to just try and use this community dashboard, it wouldn't work because thanos doesn't use the same. I think it adds a namespace or something whereas pointing it directly at prometheus does work, but I'm still not sure why this isn't working. Why oh tls skip verify, but that's what it is.

C

A

Rick for the win.

C

Yeah, thank you all right, um so our data source is configured now I can come over here to um where are my dashboards?

C

I so infrequently do this that I always forget manage, so I want to import a dashboard, I'm going to use number 1860.

C

Load, node exporter, full sure that sounds great imports. Oh right, you do need a dash or a data source. Don't you and just like that I've got my node exporter right, so yeah there's! I just stood up this cluster. You can see this morning at 7, 15 or whatever, but yeah. That's that's how we create custom dashboards right! You want! You want to fully create something yourself right. You can go in and you can create a uh a brand new.

C

Whatever metrics you want inside of here type of thing um you know: do it create dashboards right? Do all the things that you do. um I will admit that I am terrible at prom, ql and managing these grafanas. So I am not going to embarrass myself by attempting it but yeah so node exporter full. So just like that super easy. Let's change this to the last 15 minutes! Sure- and I can from here.

C

If I wanted to there's a bunch of different configuration that I can add so, for example, if we go back to our uh grafana github graphon operator github in here, so I'm in the grafana operator deploy examples directory. If we go into oauth, I can integrate this with openshift's oauth right.

C

So, instead of me having to log in using you know, when I browse to this external route, instead of me having to log in with my admin, username or a separate authentication scheme, I can integrate it in with openshift's authentication so that way that only openshift users are able to access this right type of thing.

C

So all right, so that one's good uh right we we're able to deploy it, we're able to customize it. Everything will persist because we've got a persistent volume inside of there. So I just switch back over our console and if I scroll.

B

C

To our persistent volume claims here's my kufana pvc, which is happily bound to our or happily mounted by our running working grafana.

C

So let's go ahead and destroy that guy, because that's how I do indeed so, let's come to our grafanas and this time we're going to delete the grafana.

A

And rick, I don't know if you're joking about the book uh yeah, but if you're not go ahead and plug the title in.

B

A

Appreciate your help, it was awesome that you jumped in there if you're still on.

C

So this time I'm going to create a grafana data source all right, so this is just the other side of what I just showed so before I went into prometheus again, because it's a persistent instance and provided all of this stuff manually, just like if I had deployed grafana to a standalone host somewhere this time, I am going to configure it as a ephemeral grafana instance that is going to get its config, so data source and dashboard from something else.

C

So let's go to bear token. So, let's regenerate a new bearer token.

C

Oh, I need to create our service account. Oc, create service accounts.

C

I'm typing this one. I can copy and paste.

C

And then I want to, I need to do the same thing of assign permissions because it got destroyed before add policy.

C

Now I can generate my token.

C

So I've got a new bearer token for authentication, so we'll copy that and come back over here and we'll paste this guy in.

C

So other than that, you can see that I set all of the same things in here that I did from the gui a moment ago right so yes, this is the default. I'm going to set an authorization, header tls skip verify because we're not providing the ca trust chain- and here is my bearer token. You can see that this is the secure json. That's why it was dotted out in the gui and here's my endpoint url again prometheus, so we'll click create on echo.

C

Now, if I had a grafana deployed, creating that object would trigger it to be redeployed, so it can read so it can introduce this data source, so you don't have to undeploy grafana to do this, I'm just doing it because I'm going to deploy one without apvc.

C

And so I'm not going to create a dashboard here. Actually I will create a dashboard, but I'm going to caveat it by saying so, let's go back to our examples, examples and dashboards. So we have this dashboard from grafana and what I have found is when I create this object.

C

It queries the grafana endpoint, the grafana url, somewhat frequently like frequently enough that every time I've used this method to create one, it has resulted in an error after a few minutes where grafana basically kicks back of 400, saying you're querying too much so it'll work for a few minutes and then it stops working which is not good, so my work around for that and I don't have it hosted externally. Unfortunately, I copied my that dashboard, the json for it and I dropped it into a url on my local one.

C

Actually I wonder if that'll work, probably not because I think that's a very.

C

A very plain one: uh actually, you know what I bet I can do uh download json and what happens if I open this.

C

I know right I want to see if I can put this into a gist.

A

Oh, it's going to work.

C

Plug that in there and create this I know y'all can't see what I'm doing, but I'm I'm literally just creating a gist off to the side here, all right, so I've got that where'd my window go here. Yep, I know I I undeployed you you're very broken, so grafana dashboard from url, so we'll copy this out.

C

So I'm on the grafana dashboards. I want to create a new one based on what we just copied so grafana dashboard from url, and I'm going to replace this with the url of my gist.

C

So just to show you what that looks like is this dashboard.json. It has all the stuff in here. I suppose I do need to use the raw url so we'll copy that guy.

C

All right and we'll go ahead and create our dashboard. So again I'm doing that because for me personally, I and I have not looked into it at all whenever I configure that you know from grafana.com source, it causes errors for me and the last thing that we need to do is create a new grafana.

C

So let me copy and paste the grafana instance that I had before.

C

And we'll kind of walk through this again so same stuff as before: right config, here's, our security, username and password.

C

Technically, I don't need this because in theory everything will be created for me so and then I'm going to keep this as simple grafana here, you'll notice that I'm not providing a uh data source down here, but I do need to add one thing so when I created that um I'll use this as the example, so you notice, when I created this grafana dashboard, I added a label to it app grafana and the reason why I want that is because down here, I'm going to add a dashboard label selector.

C

So it will automatically import or ingest any dashboard objects that have a label with the name app and a value of grafana in it.

C

So now I should be able to hit create here and hope for the best, and while we're doing that, we're going to check on this guy and see that things are probably still horribly broken. I can't believe that actually worked.

C

A

Time, wow, never adult.

C

I did not expect that to actually work. It came back with the name localhost, that's I. I think um let me think about that for a minute. So, let's see, let's do oc, get pod action, open shift.

C

The localhost- let's look at this guy, so oc'd.

C

Gen open shift cd dash, oh yaml, pipe 2 less.

C

So I think this worked because xcd yeah actually configures with ips.

C

I'm I I would be surprised or I I would not be surprised if we see some errors like querying the api, because it's going to have a certificate that has localhost in the name yeah, but at least etcd came back at this point. um That alone is yeah.

C

Hey. I learned right yeah, so it will technically work. Even if you screw up your dhcp reservation and don't don't.

B

A

All right, oh we did that on purpose. uh We meant to do that. Yeah.

C

100 so yeah our cluster is uh here. It's back. Everything is happy. Everything is healthy, um at least on the surface. Again I I'm a little suspect of that local host, but uh yeah you saw, I didn't, do anything except sit back and wait.

C

What's complaining over here, yeah installer control to control one, I could probably delete these and they'll reschedule. If I had to guess I'm oh yeah, these are old, created 743., I'm ruthless with these uh failed pods by the way.

A

Oh dude, I am too I I'm I'm all about. Let's see like delete pod dash all.

C

All right there we go so delete all those they haven't come back, so they they weren't valid anyways, but yeah see it was uh it's straightforward right, replacing a control plate node, even with uh assisted installer or any of the others without ipi installed there literally reprovision the node pointed out that extracted control, plane, ignition file and basically let it join the cluster and everybody all of the operators work their magic all right. Let's get back to our grafana real quick here.

C

So let's see that this guy should be done, go to our routes, open it up again. So this time you see, I am not logged in, but I should have a node exporter dashboard yay. I didn't have to do anything.

A

And the cool part is that, like all this could be automated, you know I mean like there's, there's no reason why you have to go through and and go through the ui and do all this configuration. You know you can pull down all your manifest and then use some type of pipeline or argo or whatever.

C

Yeah, you know where where's where's christian get ups get ups all the way.

A

I know I'm trying to get my t-shirt, I'm plugging it enough to where I had a free t-shirt. So.

C

So yeah it um you know again. I could uh I can log in here and do all the normal stuff, but remember this one doesn't have a pvc attached. So any of the changes that I make here are going to be ephemeral, um so I you can get the best of both worlds. I think where you attach a pvc and you use those custom resource definitions.

C

I don't know precisely how it would treat that, because if we look, if we go to our pods and we go to our grafana deployment pod, so if I do a mount and grep for dashboard, for example right it mounts these underneath grafana etsy, you know a file system, so I I don't know if those will kind of for lack of better terms, step on each other um or if the ones that you create. Actually you know end up in somewhere else.

C

If I were to create them myself yeah, you can see. There's you know provisioning.

C

I haven't investigated with the operator enough to see when I create that pvc, where does it mount the pvc at? Does it get mounted here and if so, you know when I do this cd into dashboards.

C

So you can see, oh, I didn't create the dashboard. Did I yeah.

B

C

So yeah it's one of those. I don't know precisely how that behavior works and what happens there so anyways.

C

Just like that. Surprisingly straightforward. I, I will say that I banged my head a little bit up until I I found this blog post that explained the uh the git token um command here once I had that it was uh pretty straightforward to just point at these things um but yeah I I on slack. I asked the engineering guys and it was like.

C

I asked them, and then I sat there and I was thinking about it for about five minutes, and I said you know, I should really like search for this and, of course, as soon as I searched for it and pulled up this blog post, they responded with. Have you seen this blog post.

A

Naturally, yeah.

A

uh So so thomas hall asked: um can we put all the data in config maps or crs? um So if you're, if you're gonna, do custom dashboards, that would actually be like a dashboard resource, um and then you know like so. If you had like uh some config stuff that needed to be in a map yeah, you could do that so, like the way that we do it on our patterns, is we actually create the dashboard resources and then um the grafana templates, and all that and yeah it? Just it just kind of works.

A

You know. So that's.

C

Pretty awesome so here's a example in there again in their github repo, um so grafana dashboard, that is pointed at a config map and then down here's the config map that has all of the uh the ammo for it. So you can do it that way too.

C

They have uh a bunch of different options for getting those. So here's from grafana.com from configmap from a hosted url, let's see here's one with plugins. So if you want to add plugins, you can do that um key cloak, I'm not familiar with simple dashboard here. You can literally define it just as json in the object, if you so choose.

A

Yeah, that's how we do our via json.

A

Yeah, it's pretty cool and and the fact that it's like super, easy and well I'd say super easy right, but it's it is easy ish to be able to get some metrics like that and stuff. That's like relevant to your organization.

C

Yeah, all right, so um that's that's all I got for today. I'm uh I'm gonna have to poke at this cluster.

C

In the background like after we get off the air because I'm I'm still, I I'm very suspect that this thing works like um it's: a testament to the resiliency of openshift and ncd in the control plane, um more than anything right that it came back up and we're we're technically reprotected right, if you will um against node failure uh again, I just I'm a little suspect of that and I the right thing to do would have been to reprovision the node and make sure that the dhcp and ipad dhcp assigned hostname and ip address came up correctly, which they very much did, not in my instance but yeah, and if you, uh if you are provisioning it any other way.

C

So I could have assigned from the live iso environment. I could have done a static ip. That would have worked exactly or just fine. Note, though- and I learned this yesterday- so if you do the copy network, so if I boot into that live iso environment, I set my static ip. I set my hostname.

C

If you do the copy network, the hostname is not copied across.

C

So you definitely want to make sure that your reverse dns is in place because that's where it'll look to get its host name from and interesting note, if you have ever done a uh uh assisted, installer, install right, you know in the after you get download the assisted, installer iso, you boot your nodes, they pop into the interface, and it has the host names there. You can rename the hostnames right.

C

You can give them a hostname, so the way that that works is by it uses ignition to put an etsy host a hostname file in place. So while that works, it's not the official way of assigning host names.

C

So in this instance that really wouldn't work for replacing our hostname, because we can't- I guess you could you could probably extend the master.ign to add in creating that file, but that feels kind of hacky.

C

So really you want. You want reverse dns to be the method you you want to, do it according to the documentation and all that other stuff. So, while the uh the assisted and seller folks do the the magic that they do and that type of stuff works, and it's certainly an option, I felt like doing it this way with dhcp and reverse dns, even though it didn't work for me uh was the more um it was the more common way of doing that.

C

All right, so uh any uh any last minute questions anything like that.

A

Yeah we had one come in for mike he's asking if we've ever seen any like cluster degradation over time in aws to uh resource constraints, essentially like burst limits uh for cpu and then just explaining a little bit further. Like um he's seen where the clusters are running fine, but then they become degraded over time because of the high resource utilization. So my experience has been um like whenever it happens like that, it's normally because we're not balancing like infrastructure services and application services correctly, and so what we'll do is we'll end up rolling.

A

You know we'll have customers that just don't want to do infra nodes for the cloud cost or whatever so kind of move to that cycle, especially when they start putting things like odf or service measure. You know they just want to put the registry and the router and all that stuff out of their application nodes, um then we'll see it kind of rebalance. But, like honestly, I haven't seen anything over like a long period of time. That's shown that you know it'll consistently like hit a resource limit and it didn't become ticketed.

A

I've not seen that. I don't know about you andrew. Have you seen that.

C

um No yes, and no right! So yes, but it's usually a result of the cluster getting just getting larger right where, especially with control, plane, nodes right. I start with four z4 cpus and 16 gigs of ram, because I think I'm only going to have you know a few nodes and a few hundred objects, and you know accidentally, I have a few dozen nodes and a few thousand objects um and lcds eating memory like crazy. um I've seen that happen, but like memory leak, type things or just that type of stuff, it's it's.

C

It's always been like workload related, you know, hey, we thought we were only going to have a hundred pods turns out. We've got 230 pods. um You know that type of stuff.

A

Yeah- and he clarified too, and he said that he's seen it's- he has seen it specifically in 4.7. um I'm trying to remember I it's just so it's been so long. I can't remember like that before there's something big about 4.7 when it came out, I just don't remember what it was and um yeah.

A

I think our customer, like initially had they were on 4.6, and there was something with 4.7 that they kind of held back on, um but then I think once they've gotten like four seven like 20 or whatever you know whatever it was after, like a certain addition, like everything was just working fine, so it could be a combination of the core os image that you're using and uh um in the you know the cluster version. I don't know it's weird.

C

Yeah I, I would definitely say if you haven't open a support case, um because that that is one of those things, especially if you've seen it happen more than once. That sounds like it's um something worth investigating uh and you know if support finds an issue. They'll bring it up with engineering, and you know we can get that fixed.

C

All right so uh for anybody who didn't get their question answered if you're watching this not live, if you're watching the recording, please don't hesitate to reach out to us, so you can find me at andrew.cell or via email, andru.sullivan redhat.com or on twitter, at practicalandrew, as well as on reddit, and all the various other places feel free to reach out to me at any point in time. I'll do my best to answer those questions if you're watching it on youtube or one of the others, feel free to leave a comment.

C

I've now finally figured out how to set up the automation so that I can get. I get an email when somebody leaves a comment on youtube, even though it's not my youtube account. So you know magic, nice, yeah uh so feel free to do that as well um johnny. I think your uh your information is what j-rock t-x r-o-c-k-t-x one.

A

Yep j-rock tx1.

C

Or johnny j-o-n-ny redhat.com um with that being said uh again, just a quick reminder. Next week, we'll be talking with kirsten newcomer about security inside of openshift the week after that we don't know yet, but we'll have something uh exciting planned for that week as well, so be sure to get subscribed. So that way, you know when we uh what we choose, what we end up, selecting if you have any ideas, any uh anything that you'd like to request any suggestions.

C

Anything like that feel free to reach out to us too um we'd, we'd love to uh know, what's interesting to you all uh sdn yeah, um that was it yep yeah, well, both 4.7 and uh so rick racal there um you know 4.7 was when we had sdn and I think it was uh driver issues, kernel driver issues with the vmx net, 3 nick driver, both in 4.7 and 4.8. Sadly, so all right. So thank you so much everybody for joining us today. uh Johnny.

C

Thank you as always for uh helping out and keeping me on track there. If anybody has any questions about anything again, feel free to reach out to us have a great rest of your week. Have a great and and stay safe out there um and johnny I'll leave you with the last word.

A

Yep stephanie, as always, thank you for everything that you do: uh you're awesome and rick thanks for all the feedback and and help in the in the stream and all the questions from the the crowd. You know we love it. So keep it up. Talk to you next week.

A