Ceph Cephalocon Barcelona 2019, 10 Jun 2019

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Nobody Knows What PGs are Good For, Only I Do - Danil Kipnis, 1&1 IONOS Cloud GmbH

Description

Nobody Knows What PGs are Good For, Only I Do - Danil Kipnis, 1&1 IONOS Cloud GmbH

While being a member of a team developing a distributed low-latency block-store, the author has been asked the question "why not just use ceph?" so often that he finally forced himself to understand some of the crush/rados basics.

It turned out, the block-store his team was working on was fully "declustered", while the ceph PG-indirection layer allowed to limit the level of "declustering".

PG concept is explained in various online sources but rather from a ceph configuration perspective. In this short talk, it is shown from a developer perspective on a very small example cluster what particular technical problem PG layer solves and how. Specifically, we illustrate the trade-off between cluster utilization during failure recovery and data loss probability due to coincident device failures and show how exactly PGs allow to favor the former or the latter.

About Danil Kipnis
1&1 IONOS Cloud GmbH
Linux Kernel Developer
danil.kipnis@cloud.ionos.com
Danil Kipnis is a Linux kernel developer at 1&1 IONOS Cloud (former Profitbricks GmbH). He works in the storage team designing and developing components for in-house SDS solution centered around a low latency RDMA network. He holds a Master's degree in Computer Science from TU-Berlin. As a researcher at Telecommunication Network Group, TU-Berlin, he published several papers in the area of MAC protocols for wireless sensor networks.

His most recent talk has been at Vault 2017, Linux Storage and Filesystems Conference, where he presented IBNBD - an RDMA network block device driver.

A

Hi I'm Daniel from Iona's cloud. We provide infrastructure as a service to our customers and we use a stack where we provide each customer vm on two different storage machines, and then those volumes are exported over the network to the hypervisor note where they form a raid 1, and this volume is then used by the customer VM.

A

So our squeak scale our clusters by adding new pairs of storages to our to our to our storage cluster, and then we new volumes are provided on those 2 new machines and also some old volumes have to be migrated in order to balance the load in the cluster.

A

Of course, each pair of storages is used by multiple clients and what we want to do is go toward some distributed blocks store so which what we want is that each customer volumes is stored on each machine in the cluster in order to our reach the highest throughput possible. If you do, if you go in in this kind of domain, then the client needs to take the decision for each data chunk to which pair of storages and to which pair of disk to writes its data to so.

A

If you do that, in the end, you you you have, you have two separate problem. One problem is that if each volume is spread across of all the machines, then you have the highest read: write throughput possible, but if anything breaks in the cluster, then possibly all customer volumes are affected and the other problem, which is in fact only related to the first one, is that if each disk shares a portion of its data with any other disk in in the cluster, then you have the highest research organizations throughput possible.

A

But you have a high probability that another disk fails during the recovery. So there is P. Gene Direction layer addresses exactly that problem and it's it's introduced in those famous self related papers. I want to illustrate it on on a small example: cluster, consisting of just three machines each having two disks and the replication policy being the rule would be to put each block of data on two different disks on two different machines. So now there is a question arising like how to organize the disk into pairs.

A

The simplest would be to just group each two disks in into a pair, and then the client can write into such a pair. This is what is called a fully clustered setting. So basically, in order to recover one single disk, if it is, would be failed to recover the data. I need to write from only one disk in the cluster.

A

So of course, one would like to be faster so and maybe want to forage this to participate in two different groups and maybe even more first and it could participate in three different placement groups and then, finally, it could participate with four other discs, which is the full. It is clustering setting for this particular cluster. So here, when I need to recover disc, a then I would I can read in parallel from from different disks.

A

But then again, this particular example presents a trait of a special trait of which entry, which is introduced by the number of placement groups. So, on the one hand you have the highest recovery speed. On the other hand, you have the high probability that some other disk fails during you recover a given one. So if you look into this example cluster you, you would see that we have six disks.

A

So if you draw exactly fifteen possibilities to select two discs, two random disks out of six, but in that with three, we only use three placement groups. So so that's obviously like if two disks random G's fails, then we lose customer data with probability of 20%, that particular cluster, and if he had six groups and would be 40% that if we have nine groups, it will be 60% and twelve groups the full G clustering setting. It will be at 80% and of course, in general you can.

A

It looks like that, which is pretty obvious, so I'm not going to spend more time on this one, and but my point being, is that the PG indirection layer is a very general concept, so it on the one hand, it provides a level of control of kind of data distribution. So on how many physical devices a volume, a final volume is distributed.

A

On the other hand, it provides the level of control of the de clustering, so how fast I can recover one failed device Oh with how many other machines or drives a given drive, is sharing its data with, and it's a general very general framework for implementing key replication policy or cluster scaling and as an example here is an example. On the left side, you can see a very familiar to you picture of kind of RPG setup. I have RPG images on top it's they sit on top of Radice objects.

A

They are mapped while some hashing to placement groups and those placement groups are stored on on OS T's. On the right hand, side you see a stack built with using just DM thin and each customer volume is each volume thin.

A

Provisioned volume is exported with some network protocol like SRP IP in video, and they Envy me over Fabrice to the client they built into rate arrays on the client side and then, finally, you can put array 0 on top and you can manage this kind of configuration using same approach as crash is doing so in theory, you could use, you could use a safe monitors and get the configuration for for this kind of stack and but the main challenges of doing so are those and now I'm done thanks.