Ceph Cephalocon Barcelona 2019, 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Healthier Ceph Clusters with Ceph-medic - Alfredo Deza, Red Hat

Description

Healthier Ceph Clusters with Ceph-medic - Alfredo Deza, Red Hat

ceph-medic is a small project that helps identify issues with a Ceph cluster that may be difficult to detect, even when using automation. The project lead will demonstrate how one can quickly discover issues with a running cluster, regardless of the deployment type (bare metal, containerized, or kubernetes), emphasizing on the importance of good reporting, clarifying errors or warnings for better deployments.

About Alfredo Deza
Redhat
Principal Software Engineer
Alfredo Deza is a principal software engineer working for RedHat on the Ceph distributed storage system, avid open source developer, unit test enthusiast, Vim plugin author, photographer and former athlete.
As a passionate knowledge-craving developer he can be found giving
presentations in local groups about Python, file systems and storage, and
system administration. Currently leading the development of ceph-volume, ceph-medic, and the build and release infrastructure for Ceph.

A

Hello, my name is Alfredo Lisa I work for a Red Hat and we're going to talk about healthier, safe clusters with set medic.

A

So we have a very good QA team and but sometimes developing the the volume tool we we will get into deployment issues that were really really tricky to to understand. Where we're coming from a like. We deploy this cluster we're using our standard deployment practices and things are not coming up like what's going on, so please come and help us, and usually that means say here's a bunch of notes and we have something that didn't come up.

A

So what do you think it is, and so, at the first at first you go and check all of the all of the different items and number one in the list is the the Rados troubleshooting guide because we're deploys these and things are not coming up. So you go look at the logs, you might be checking if system D is doing something funny or well. You go through all of the different steps in available in the troubleshooting OSD guide. You know, like DF, see CTL set.

A

Yes, what's what's going on with this cluster, and why why why are things are not exactly coming up, so issues can be really really sneaky like it's. In in our experience like, they were not immediately apparent like why things are not quite working, you know like it's not like I know, it's didn't come up because SEF was crashing, it was, and this is usually why the QA team would be like hey.

A

Can you please give us a hand because, like we really don't don't know what's going on so one of these situations, for example, was Hades. Osd is not coming up and we don't understand why, and so it happened that the key for the OSD, for that particular OSD, was completely different from what the key that the the authentication key they had to use so somewhere along the step, someone changed a parameter and that particular OSD in the hole in the whole cluster changed and couldn't authenticate.

A

So he wasn't coming up, but nothing in the lots is going to tell you well, you know: I tried to come up and I'm using a different key. The cluster is expecting D. So that's. Why and you can you can understand that the first time, but the second time around you probably forget it. It's like another tenth when you know cluster and you're like well, I, don't know what's going on so what? What do I do all right, I, let's check the key on the on the FS IV or on the LSD.

A

Maybe that's it, and sometimes it isn't and and- and you know it's one thing after the earth, so we started out like writing these very small checks. So let's go through that list of like 20 things that we know like right away. So we don't forget and doing that is automating automating. That error checking process is, you know it removes the being error-prone, make it fat you make it fast and then you don't forget things. So we made a very small tool called somatic.

A

This might look like super tiny, but it doesn't matter so somatic is the thing that you can just point out your cluster. It has support for docker containers, openshift, kubernetes or bare metal hosts and what it does is just go scene and contacts all of these different nodes in your cluster and it's gone. It has the context of the whole cluster. So it's not like. Oh, this one node is causing a problem but like what's was a difference? How can I compare that information to the rest of the cluster, which is really important?

A

So, for example, we were recently running against a cluster in kubernetes and we found out that the cluster FS IV in one node was different than the other nodes, so things were not quite as expected. So there was this one. No, that was not joining the club. The cluster was like I, don't know what this cluster. What this other node over here is doing, because all right I don't recognize that FSIA.

A

That's not me, so this is very tricky to find out if you're going to manually, go and poke in at each using every single server or every single pot or container. But it's really easy if you're automating- and you have this tool that suddenly it can compare, can compare things around. So you can run again to anything like I said containers. He has support, be a uncivil host file, so you just look at them and just connect. Oh, you have four: is this year monitors over there and it works?

A

Pretty smoothly- and it's documented here- docks at safe, calm, /, Seth Matic how to install a how to run it and I hope you guys, try it and send some system tickets and some issues, so we can improve it. Thank you.