Red Hat OpenShift OKD Working Group, 17 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OKD4 Deployment of Ceph and MariaDB Galera Cluster Charro Gruver Red Hat OKD Deployment Marathon

Description

OKD4 Deployment of Ceph for MariaDB Galera Cluster
Charro Gruver Red Hat
OKD Live Deployment Marathon
August 17, 2020
Day Zero Kubecon Eu

A

Okay, so a lot of a lot of what you're going to see here, I pulled the configuration that we're going to deploy directly out of the rook project, which you should be seeing on your screens right now.

A

So so what you see me um running here, I I pulled from here specifically.

A

Let me clear this out. If you guys can see the screen on the left, the um common yaml file and the operator openshift yaml file, those are pulled verbatim from this project. You can see the you can see the path here to get to it and it's release 1.4.

A

It's the latest release, I'm not quite brave enough to run directly out of master at this point so uh back to the cluster that we deployed earlier this morning, um you can see it is fortunately still healthy, um there's a few errors being thrown, but it tends to do that in my home lab network latency and such so. What we're going to do first is.

A

The three worker nodes that we deployed- what I didn't tell you when I deployed them, was that I actually deployed them with an unused uh hard drive attached to the virtual machine, so it installed the operating system, uh it's using a sata bus, so it installed the operating system on sda, but it's got an sdb sitting there that is not currently being used.

A

What we're going to do is we're going to create a ceph storage cluster to serve up block devices on these worker nodes. The first step is, we need to label those nodes uh to give them a roll of storage node, and so I just applied that label to them. If I hit a quick oc describe on one of those nodes, I can show you that it now has a role: a storage, node, okay, so that's step. One is we. We need something to tell seth uh what it's going to be working with step.

A

Two is we're going to deploy um this common diemel, which, as you can see, is creating a whole lot of boilerplate that the rook operator is going to need, and one of the things that it did was it provisioned for.

A

Us a namespace, the rook seth namespace, which currently is very uninteresting, we're getting ready to make it interesting by deploying the rook operator.

A

And this will take a little bit for it to bootstrap itself. The the operator image right now is pulling down and installing, when the operator is up, it's going to create uh some workloads, some pods on each of those nodes that bear the label so that they can discover the the resources available on that node. And this this will take just a little bit to run.

A

Okay and there you see the the three uh discover nodes that are spinning up now,.

A

And while those are coming up, let me show you a little bit bigger so that you can see it on the screen.

A

Okay, the cluster.yaml file is the thing that actually defines um our particular ceph cluster and again, the the um rook project has a a boilerplate copy of this for you to to take and modify to your own purposes. This is the version of ceph we're going to be running 15.2.4.

A

We're going to have three monitor nodes running. I have set node affinity on those and that it's also going to be looking for a role of storage, node, I've assigned resources for the various components of seth, uh a limit and a request.

A

Just like you see in a typical deployment, and here is the piece of magic that tells it where to find those devices that it's going to create the stuff storage cluster on now, our operator appears to be fully bootstrapped and up and running, so the next step is to go ahead and deploy our stuff cluster. On top of that all right now, this is also going to take a little bit and you're going to see a bunch of activity here. As the operator provisions this cluster there's the three monitor instances. You just saw spin up.

A

There's the csi plugins.

A

Here in a minute, it will start actually dealing with those physical devices and formatting the storage for its own use.

A

Okay, you see these osd prepares, there's three of them when they are done they're going to go into a completed state. Once you see that completed state.

A

Then the seth cluster is up, and it is ready for use, uh looks like we're still waiting for one of the crash collectors to go into a ready state, but everything else at this point should be usable.

A

So to prove that it's usable, let's take our image registry and let's give it a persistent volume.

A

I've created a storage class here that I'm going to apply its name is rook seth block, and it's going to use the new rook um csi plugin that we just deployed as the provisioner.

A

So I will apply that to my cluster.

A

Right now, if I flip back over here, go down to storage, we should have a storage class, and indeed we do now.

A

Let's create a persistent volume claim, 100 gigabytes, because I tend to use a lot of container images.

A

All right now we should have persistent volume claim and the key here you can see now it is bound to an automatically provisioned persistent volume that our ceph cluster kindly handed out for us. We can see that from the command line as well there. It is 100 gigabytes now, remember if you were watching previously when we deployed the openshift cluster, we gave our image registry an ephemeral volume.

A

We need to remove that ephemeral volume before we give it. The new volume so caveat here. Any uh images that you current that you had put in between you you're going to lose those because we're yanking away the the storage. uh You will lost them anyway, because this is an ephemeral.

A

Volume: okay, now going to put our.

A

Registry back into a managed state we're going to tell it to use the persistent volume claim registry pvc we're also changing the rollout strategy of this to a recreate, because I created a rewrite once volume.

A

um The the rollout strategy that comes by default is not going to work because it it will try to to do a rolling deployment.

A

I need this to tear down the first instance and create a new instance so that it doesn't try to violate the read white read, write once policy that we just patched it if we log back into our cluster and look at image registry.

A

Here we go, we have an open shift image registry, that's creating and should be binding to that persistent volume claim.

A

I'll pause there any questions on that. I know that was pretty fast.

B

I think you're doing pretty good here.

A

All right, I will now deploy a maria db galera cluster.

A

It's going to be a stateful set.

A

All right, what we're going to do we're going to deploy this stateful set right here, which is going to create a three node maria db galera cluster, from a customized image that we're going to build and push into the image registry that we just created.

A

First thing we need to do is build our image.

A

Which the official mariadb repository is going to install uh 10.4.

A

Our docker file looks something like this, so we're going to do some things to set up a mariadb. The real magic happens in this.

A

Shell script, that is going to be run by the image when it starts up that actually provisions the mariadb cluster detects whether or not a cluster already exists, if it's the first node in the cluster so forth, and so on. I've got a short tutorial written up on this that you can see. So I won't drain it here. We'll just do the fun and kick it off.

A

So first thing I need to do is make sure I'm logged.

B

A

Okay- and I am in the right cluster- important safety tip- always make sure you're logged into the right cluster, so I need to expose a route for the image registry. What I just did is I patched the image registry operator to say: create a default route so that I can externally get to my image registry.

A

Then I'm going to use podman and I'm going to log into that image registry array. It succeeded and now I'm going to do a podman, build and build our mariadb image, and you see I'm I'm grabbing the route from the from the image registry to tag my image that I'm getting ready to build so that I can push it to the directory.

A

And it generally doesn't run that fast. I ran this just a little bit ago to make sure it was going to work. That's why that build went so fast because it had already actually been built.

A

So all it did was add a tag to the already built image.

A

Now we'll push it to the registry.

A

Okay, so now our openshift cluster in its local cluster registry, it now has our customized mariadb galera image.

A

Let's create a namespace for.

A

It, let's create a service, account: okay, we're going to create a service account for mariadb. The reason is: mariadb is picky about its uid and it's especially picky. If it restarts and its uid has changed, it tends to get upset so we're creating a service account and we're actually going to run this privileged uh with this new service account so that it can run as any uid.

A

There is likely a better way to do this, and so I am open to suggestions and now I'm going to apply a config map that.

A

Actually contains the mariadb server.cnf file by using a config map. To do this, I can modify the configuration of my cluster without having to deploy new images. I'm now going to apply a couple of services.

A

One of them is a headless service that allows the cluster to talk to itself on the necessary tcp and udp ports, and then I'm going to deploy a load balanced service that will allow applications to talk to the cluster.

A

Not sure why that's taking so long to come back! Okay, now before I hit this, I'm going to switch over here so that you guys can see the deployment all right. So I'm going to deploy the stateful set and what you'll see in the console is a ordered deployment of the mariadb stateful set.

A

Good, we should see this pvc okay, good pvc bound. Now we have persistent volume.

A

And you see the uh first node in our three node galera cluster is now starting.

B

So just pause just for half a second someone's asking frank's asking a question: he missed the very first part of your rook operator installation, and he says he has no rook operators with his ipi based cluster installation.

B

Is the operator hub filtering operators based on upi ipi installation choice? Maybe clarify that for him. Okay,.

A

Good good point: I actually deployed the operator from the command line because it doesn't show up in the operator hub.

A

See not there um not sure why.

B

It may be one of the ones that um we need to, as a working group add in a community.

A

B

There's a community version of it on operatorhub.io that works with generic kubernetes, but I think we need to do some work. I think it's one of the ones that's on the priority list for us to work on as a working group.

A

Yeah and that's why I I deployed it by using the um operator configuration that is provided in the kubernetes stuff examples in the rook project itself.

A

All right, so our second uh cluster node is coming up, and these do and these do an ordered, startup and an ordered shutdown so that you can gracefully, stop and start this cluster and it will retain its state, and when this is done, we have a three node maria db galera cluster. That is a full multi-master database cluster running in our openshift, with provision, storage, wow.

B

Well, played thanks charo. That was pretty awesome. um I think I I and I keep emphasizing um in the chat too is well. Some of the operator work is the next things on the road map that we're trying to get um folks to work on so getting some of those um default operators from operator hub into community.

B

So we will be working on that. So you fill the time very nicely justin pittman's here and he is going to try and outdo you on bare.

B

B