Ceph Ceph Code Walkthrough, 1 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ceph Code Walkthroughs: RADOS Snapshots

Description

Every month the Ceph Developer Community meets to discuss one aspect of Ceph code, spread knowledge of how it works and why it works that way.

This month we're joined by Samuel Just on RADOS Snapshots with Q/A.

Find future Ceph Code Walkthroughs: https://tracker.ceph.com/projects/ceph/wiki/Code_Walkthroughs

Ceph Code Walkthrough Playlist: https://www.youtube.com/watch?v=nVjYVmqNClM&list=PLrBUGiINAakN87iSX3gXOXSU3EB8Y1JLd

A

Hello, everybody and thank you for joining us for another stuff code, walkthrough march 23rd, 1700 utc a lot of daylight saving time to standard time changing so, hopefully people make it into this. One there'll always be a recording as well after we produce these, so that will be posted to the youtube channel, and today we are joined by samuel, just who has been nice enough to take the time, even during uh the development for pacific, uh to go ahead and take time to explain to us afraid of snapshots.

A

I do believe if I remember correctly, sam, that a lot of this back in the time, at least when I was looking at radio snapshots that was around something with a context that gets pushed around to um uh with relation to other I o activity. That is happening. I don't know if snap context is still a thing, but first you're there yeah. I think that was back in 2009, but uh so anyways, but I I will shut up now and let the expert of course speak. So everyone here is samuel.

A

Just uh speaking about rate of snapshots. Thank you. Cool.

B

All right um so uh with no particular fanfare. Okay,.

C

So I've got kind of like a presentation.

B

With some code snippets, but I've probably only got 20 minutes of content, so please by all means interrupt me. If you want me to go deeper on something, um if not, there will certainly be time at the end for questions and such all right so rate of snapshots. At a super high level, there are three or three south projects: rgw rbd and zfs.

B

Two of these use snapshots heavily um both rbd and cfs, make heavy use of them, rbd at a block device level and ffs at a subtree level, but just as they both use libretos or the object during ss case to mediate io. They use the same underlying machinery and liberators to do snapshotting as well, so for cephafest. This looks something like this, which I stole um from greg's presentation.

B

You can go into the dot snap directory of, I believe any directory in this ffs tree and any directory you make in there will under the covers, create a snapshot at that point.

D

B

Which you can then access as a read-only copy of that subtree for rbd snapshots? We have uh rbd snap commands that create list and roll back snapshots, and you can mount snapshots read only so. The two common features here are that in both cases we have a large related set of rados objects that share the same snapshot, lifecycle and uh creation time.

B

The two major design goals here are cheap snapshot, creation and cheap snapshot reads.

B

um So, at a really high level, clients maintain and arbitrate snapshot metadata for object groups. Externally, it's ffs in its internal metadata.

D

B

Ordering via the mds and capability system and rbd, maintains its snapshot. Information on the head object of each uh of each rbd block device with ordering via watch notify it locks.

B

Osds maintain per object, snapshot to state mapping, updated with snapshot, information, piggybacked on rights and I'll talk about that in exhausting detail. um Rollback must be performed per object. So if we go back to this uh rbd example, rvd snap rollback is uh requires time, linear in the number of objects in the rvd image snapshot. Space deletion is fast, but the space reclamation part is lazy.

B

Is a lazy background operation that happens across the cluster all right, so the intuition here is that let's say we have in this case, let's say: let's say it's an rbd image with five options set up. So I guess this is a 20 megabyte rbd block device with five four megabyte objects up at the top. Here we have the client state which currently has no snapshots, so the snapshot sequence is zero and no snapshots have been taken.

B

So let's say we take a snapshot at this point. The client state now has a snapshot. Sequence of three and its little snaps vector has just three and up y three and not one because snapshot ids are unique and uh increasing, but not necessarily depth for reasons we'll get to later.

B

When we perform a write, the osd is going to say: oh three is greater than zero, so apparently a snapshot has happened since the last time. I saw it right on this object, so I'd better make a clone. Oh crap. This is actually wrong. I have to do it later. No.

B

um The head object here should say three colon three that hopefully gets fixed in a subsequent slide. So let's say we take another snapshot 10 in this case we perform a right. ah The left, one here says head three cool and three that happened with the initial right. I just messed up this slide um so same deal here, except that in this case, instead of there only having been one snapshot that happened. Two of them happened, so we create a clone at snapshot: sequence, 10 with covering snapshots, 10 and 3, and the.

C

B

Object gets updated with a snapshot sequence of 10 with only one clone at 10., so we take another snapshot 19 this time and we reform it right back to the first object.

B

So once again, two snapshots have happened since the last time. This object was written, but this time it was 19 and 10.. So the head object gets updated to a sequence number of 10 with two clones at 19 and 3..

B

So let's say we need to perform a snapshot read at snapchat or read at snapshot3, perhaps because we've mounted a this rbd image at the snapshot corresponding to step by e3. These red objects are the ones that will be read. If we direct a read to the first object.

B

The osd internally is going to direct that read to the first clone we took and ditto for the others at 10 at snapshot. 10, we get these objects and at 19 we get these note for the objects where we haven't seen any writes. Since the snapshots got taken, we performed the reads it had.

B

We have any questions here. This would be a good place to put it if there's something unclear about the sort of macro structure.

A

I don't see any questions in chat so far. Yep.

B

I just figured I'd pause, okay, so the underlying interface that rbd and stuffs are using for this is the uh liberato snaps interface.

B

There are four basic operations here: snap create which corresponds to these things, where we updated the clamp information snap or set snap right context, which also happens as part of that process.

B

Snap remove, which is how we remove a snapshot from a pool and snap rollback, which rolls an object back to a specified uh sequence number.

B

So the snap create process is essentially a monitor operation. All that happens when you create a self-managed snapshot. Is the monitors give you a new snap id that hasn't been given to anyone else on that pool, and that is greater than any snap id you've gotten in the past.

B

The way this is arbitrated is pretty simple. The osd map, pgpool t structure, has a snap sequence and a snap epic that track the most recent snap sequence number given out and to ensure that we don't give out the same one twice so, every time you create one of these snapshots, it gets.

B

We get a monitor, commit that updates the maximum and you get back through the result. But at this point no actual objects exist with that snapshot. Sequence number and nothing actually has happened other than that the monitor will never give out that number a second time.

B

The second important piece here is the set snap right context, so this state and the client here sequence 19 snaps 1910-3. This is the snap right context.

B

If you have a look at the mosdop message that gets sent from the clients to the replicas on right, there are two fields here: snap sequence, corresponding to the seek 19 and snaps corresponding to this 1910 three.

B

This is the mechanism by which the osd learns. What snapshots have happened on an object since the last time a write served on that object.

B

Within the osd we have these osd level. Data structures attached to each hat object is a snapset corresponding of a bunch of fields, but the most important ones here are the map idt, which is the highest snap sequence. That object has seen and the clones vector, which is this 19 3 thing.

B

We also have a few look aside: structures like clone overlap and clone size that track the byte ranges that overlap between adjacent clones and the size of each of those.

B

Clones, this gets encoded into the snapset attribute of the head object and it gets cached in the osd as a snapset context, a bit like the object context, if you're familiar with it as we process or write, the op context has a new snapset field that projects the to be committed snapshot until the write commits.

B

So, at a high level, the osd right path looks something like this. We receive an mosd out from the client, we queue the app for the pg. When the op comes off the queue we lock the pg, and then we perform some work to package it for persistent storage, mainly constructing a pg transaction.

B

um For snapshots, the most relevant bits here are finding the current object state and the execute context make writable con combination, uh which we'll talk about in just a moment.

B

Once we have this transaction, we send it to the replicas and to local persistent storage, but that part is, after we've done the transaction bit and it already encodes all of the information required to construct or update clones execute context.

B

This gets called as early and do up, and it is the primary entry point for the primary processing a a reader write quite early in this function. If you have a look you'll note, we have a look at the get snap sequence and get snaps fields of the mosdop.

B

And we fill in some off context uh the op context, snap context um to track the pending snapshot operation. uh If it exists note, this bit only happens on uh mainly on on right.

B

This filter snap context, all filters out any snapshots that have logically been removed since the operation was was.

B

Sent later after we've already gone through the uh the op itself and had a look at all the individual operations, the last thing we do before we well one of the last things we do before we send it to uh before we complete the transaction is we call make writable and what this does is it compares the snap context we constructed just a moment ago, and it compares it to the structures already on disk. As in the diagrams, I uh showed you before.

B

If the snap context is newer than the object, then the sequence already on disk and has new snapshots, we create an object context with an object info t for a new clone and we add to the transaction and operation cloning. The object in make cloud you'll also notice. The clone gets its own pg log entry, uh the pg log, isn't actually aware of snapshots. As such, it directly represents each new clone created with a special operation code for clone.

B

We also encode into this log entry, the set of snapshots that are stored on that clone and if you stop and think for a moment at the time that the clone is created, we already know all of the snapshots that that clone will be used for, because you can't go back in time and create snapshots that happened before this point, so that set will only shrink. It will not grow.

B

So now, let's talk a little bit about snapshot. Removal deleting snapshots is also essentially a monitor operation.

B

The osd map contains these uh removed snaps cube structures that encode the total set of snapshots that are pending removal and the ones that were newly removed as part of the sosd map or part of this fabric, each osd as it consumes. These osd maps updates each pg with a p, with a snap trim cube of snapshots to remove there's a tricky component. Here, though, because snapshots apply to many objects, not just one.

B

We need a way to map a pool snap id to all of the clones within that pool that store that are kept around, because that snap is still alive.

B

To do that, we mana we maintain a look: a site index in persistence on disk called the snapmapper, um see sourceocsnapmapper.h for more information. um It gets updated online as part of uh primary log. Pg log operation update snapmap.

B

This snaps field, piggyback an hpg log entry, is how the primary and the replicas I'll learn about these updates when you process a pg log entry with one of these snaps fields, populated as part of the transaction being sent to local persistence. We also update this. The snapmapper.

B

The snapmapper, provides a reverse mapping from snapshots back to clones affected by that snapshot.

B

Which is how the asynchronous snap trimmer works as the osd advances its osd map, it populates this pg snap trim queue on each pg and asynchronously. Each placement group, as the p as the snapshot queue is not empty, will go through each of the objects in the snapmapper via getlex objects to trim and for each of those objects it will check to see which snapshots are still alive, because the removal of the snapshot does not necessarily mean the clone gets removed.

B

There may be other snapshots represented by that clone, but if there are no longer any live snapshots, the clone gets removed and either way we update the snapset on head with either the reduced set of snap them snaps or with the removal of the clone entirely.

B

uh The prime. The driver of this is the snap trimmer state machine in primary log.

B

Pg reads are relatively simple: um do-op when it is trying to work out which object to work on calls into find object. Context with the snapshot, sequence of the op that got sent and it maps the snapshot sequence requested onto either the.

D

B

A clone depending um note that the behavior here is not dependent on the snapdrummer, whether the snapshot, a snapshot that has been removed will be detected as removed, whether the snapdrama has gotten to it or not because find object. Context will check the current osd map for whether the snapshot is actually still alive or not.

B

The last little bit of information is recovery, but there are a few important bits of recovery that have snapshot related touch points.

B

um The first is that we need the snapset attribute on the head in order to work out which clones are supposed to exist and some other things so before we recover a clone to the primary. We first recover the head object and you can see that in replicated back end recover object.

B

Secondly, because the clone operation in object store that we use when we get a write on an object that needs to be uh that needs a clone created, is meant to have um copy and write semantics, where the underlying persistence shares extents that haven't been modified between those two clones.

B

For that reason, it's fairly important that recovery reserve that data sharing or else after recovery, you would find that a pg uses up more uh space than it should so. For that reason replicated back end and there are analogous there's analogous code in the eraser coded backend, I believe, use these uh calc clone subsets and calc head subsets to compute ranges of the object that can be cloned from adjacent clones. That are already present.

B

And I think that's all I've got for this presentation.

B

Yeah. Do you have any questions? I can go into more detail. I have an emax window up if people have questions.

A

I don't see any questions listed here, but I had one in particular uh for myself in terms of uh originally, I was wondering how like from a client perspective, if somebody was to actually be interacting with it. Well, is this api actually available directly to radius itself uh to be making any sort of snapshots like this.

B

So there are two versions of that question. The first is no um self-managed snapshots, as you saw require a great deal of cooperation from the application code. So hypothetically, you could create a liberators, an application that uses libraries directly, in which case yes, you can use self-managed snapshots, but the way you interact with self-managed snapshots normally is through the rbd snapshot mechanisms or the cfs snapshot mechanisms.

B

There is a totally separate mechanism that shares almost all of these underlying implementation. Details called pool level snapshots.

B

And if you look at the step documentation, I think these are even still represented. The trouble is that they aren't useful for very many things. You certainly wouldn't use them uh they're mutually exclusive, with self-managed snapshots, so a single pool can't use both.

B

So if you do use a pool snapshot, you can no longer use most features of stepfs or rpd on that pool, but, more importantly, even with rgw or something else or some other application, they don't behave the way you would expect um a feature that has the word snapshot to behave really, essentially because they aren't point in time. Snapshots they're, somewhat complicated and their state is gossiped along with the osd map, so in general, that feature is fairly deprecated. I believe greg is that right.

C

I I don't know if we've actually like written that word anywhere, but yeah don't use there.

B

There really aren't any circumstances there. It would be hard to imagine a scenario in which it would be the answer to a question you're actually asking.

D

B

A

In short, yeah.

D

A

B

You self-manage snapshots is through projects that incorporate directly.

A

Yeah, the other components got it.

C

A

Sounds good well um if anybody else has questions, while sam walks, through the code, feel free to throw it in and I'll try to catch sam at the right time, so he doesn't have to monitor the chat window. At the same time, thanks everyone.

E

I add a quick thing: we can also cover it uh when you're walking through the code. Are there any quirks about um snap trimming which make it um different or like? Is there uh anything special about snap running uh versus when we do regular, pg, deletions and stuff.

B

I'm not following the two mechanisms are essentially unrelated.

E

I mean uh I am pointing this out because you've recently seen some reports uh of performance depreciation uh with our blue fs buffered io change, which only show up um in pg deletion case and snapshot. Trimming case.

B

E

That is true, but like can we categorize them as just like in the same kind of work, or is there anything special about snapdragon, shot trimming versus pg deletion, or can we just say it's the same kind of background work um or not.

B

Snapshot trimming requires locking pg, though, that contention could be worse.

D

B

My guess is that it's private, if, if, if the behavior was caused by or was exposed by blue store level changes, my guess is that it's the I o itself. So yes, in that sense, they should be pretty uniform.

E

Okay got it thanks.

B

I wasn't actually going to do any further code walkthrough. The reason why I assembled the slides was that I could pull out the relevant bits of these functions as the functions themselves are very long. um I can pull up. Is there something in particular people would like to look at?

B

I can pull up find object context. This is uh fairly instructive. I didn't put this one in because it was too difficult to fit it onto a slide.

B

The important bits here are.

B

B

This oid here this is constructed out of the components from the mosdop. um It hasn't.

D

B

Been mapped to a clone, that's what find out context does so if we scroll down a bit past that we're here.

B

This covers the case that we're actually looking for the head object. This is the normal case that you perform an ordinary reader write on the object. If you're performing a snapshot read the snap field will not be no snap.

B

um This map stop id to clone is a special path for uh cast sharing. Unfortunately, this part models, the behavior- I showed you before, where, if the snapshot sequence is smaller than the sequence being requested, it's necessarily a request for head.

B

This is the portion where we walk through the snapset to find the correct clone for the object. um It will be essentially the first clone with a uh with an id greater than or equal. I believe to the to the snapshot requested and this portion here at the end validates that the snapshot in question actually maps to that clone by checking the uh clone snaps field it snaps up.

B

um This could happen if, for instance, you asked for a snapshot that was never mapped to this particular object. Then it won't fall into that range and you get nad instead, uh potentially the behavior. Here we take a little bit more look at the recovery code.

B

D

Really quite a lot to cover.

B

Here so any oh, a question got asked: does clients need to know snapshots to talk to rados? How do rbd and cefs clients get the correct snapshot info after a snapshot is created, so those questions are slightly different for rbd and zfs. um I'm going to tell a story about the way rbd works, that if jason wants to correct me, that would be great at a super course level. What rbd does when it creates a snapshot?

B

Is it requests a new snap id for one thing: rb is creating a snapshot on a block image, so it already has, I believe, a rados level. Advisory lock on the rbd head object for that image, and so the first thing it does is it asks for a new snap id from the monitors on the relevant pool.

B

It updates the metadata stored in the rbd head object for that image, and then it issues a notify notifying all other users of that head object to reload the metadata corresponding to snapshots. For that object.

B

um There is an authoritative updater, I believe, which is uh the holder of that advisory. Lock, the other holders will be um rbd clients mounting read-only snapshots of that image. I believe, does that answer. Your question. Cfs has a rather more complicated mechanism involving the cap system, but either way the application that is using self-managed snapshots is responsible for making sure that the snapshot information gets propagated, though in both cases these liberators mechanisms to do so does that make sense.

D

Yeah, it makes sense cool.

B

That's one of the reasons why there's that there are no high-level command line ways to access this, uh the assumption or the the key to this system is. There must already be some kind of right ordering or metadata system overlaid on top of liberators, so attempting to create a snapshot level metadata would be wasteful, it's more efficient to integrate it into whatever metadata already exists.

D

And the clients really have to have the correct snap snap set to talk to radars right if they use something really bad. Something really bad happens right if they use something wrong. That depends.

B

um If a client does not know about a snapshot that does exist, then what will happen? Is they obviously can't be trying to read that snapshot, because they don't know about it right, so the osds will see snap contacts come in that are out of date. They'll go ah well, I'm going to ignore your snap context and simply give you the snap id that you actually want, and in that sense it will still be completely correct right.

B

If I ask for snapshot 5 and in fact we're up to snapshot 30, then the osd still know how to map snapshot 5 onto the relevant object. Even if I don't know that the later snapshots exist so for reads: it's a non-issue for rights: there must be an authoritative writer. Essentially you won't kill the cluster specific state.

B

You still get a well-defined answer like if you have two different, um authoritative or if you have two different rbd writers that are both trying to create snapshots in the same blog image and sending rights to the same in to the same image.

B

In addition to the fact that it will no longer behave well, normally, you will also get sort of inconsistent. You you'll get snap contacts that are different between different objects in the same uh image.

B

The osds won't really care, but you won't get answers at the rbd layer that makes sense to rbd.

B

That was confusing, but I think the answer is, generally speaking, the osds won't care for reads the application: won't care either, but for rights the application is responsible. If it wants to maintain it's a snapshot system for making sure that there aren't divergent snapshot. Metadata is floating.

B

B

To go, find the calc's clone subsets here, okay, so this is a little tough to read if you're not following along, but the idea here is that, because the snapset contains these uh clone overlap, fields which are interval sets um where any offset contained in the interval set for clone idi is an offset shared with clone, I plus one or head. If that's the, if it's the last one so by.

B

Looking at the clones on either side of a particular clone, we can work out which subsets we can do a local clone range on when we're recovering the uh the other clone.

B

A little bit tougher, you repeat it go ahead, can.

E

You repeat that.

B

um Yeah, so this code doesn't really lend itself to going through line by line, but the notion is: let's say we have three clones, one, two and three: if we, if we already have one three local but we're missing two, then we can look at the clone overlap, field and infer o clone two shares the for its first megabyte with bone, one at its third megabyte with clone three.

B

That way, we don't actually have to pull that data from a replica. We can do a local clone range on those regions from the objects we already have stored locally.

B

Not only is this a save in bandwidth from replicas and save on replica reads: it preserves data sharing between the addation djs and clones at the blue store level, thus reducing overall space usage.

B

So it's a pretty important optimization. This.

C

Is embedded into.

B

The into the recovery process in general.

B

Any other questions is there anything that you'd like a more granular walk through one.

A

Well, sam, you did go over specifically in terms of being able to find an object. You went over in your presentation some of the delete operations as well the trim operations. I don't.

D

B

Those are worth going over yeah, so the important so when you actually do a snapshot removal, so let's say in rbd. If you remove an image uh snapshot, rbd internally is going to remove that snapshot from its own metadata.

C

B

Issue a command to the monitor, uh removing that snapshot. The monitor is going to update its pg pool.

B

B

Remove snap seal this one, I believe um in the next osd map to as a broadcast to all osds that the snapshot is now removed. Those osds will eventually receive an osd map in the way that posties always do.

B

And activate so uh yeah activate happens on activation, there's a corresponding bit of code anytime. We process a new osd map, I'm forgetting the location of where it initializes the snapdragon queue and the purge snap set as long as these things are not empty.

B

There's a background operation called the pg or the primary log. Pg snap trim snap.

B

B

The main states, it's um structured, the same way the peering state machine is with one of these blue state chart state machines. um The initial state is unsurprisingly not turning, and the other states are trimming whereby it's going one object at a time through and uh let's see, if I get the more important one.

B

The weight trim, timer state, I believe, is where it's waiting for the there's, a configurable that.

D

B

An artificial delay between different operations between uh drum operations. In the background into this background process, um I believe that's modeled by this weight trim timer state.

B

Wait our w lock waits for getting the right lock on the objects that need to be trimmed specifically on the head. I believe weight wrap ops weights for the.

B

Weights for the replication operations to complete on the replicas, a trim operation is essentially identical in structure to a client right or any other logged operation.

B

Just as with a client right, we create.

B

Log entries and send pg transactions and log entries to the replicas, um so recovery and uh failure work the same way so with the normal write, actually not sure what async the way it is work does, I believe, a weight. Async work waits for the uh local operation to complete.

B

Might actually be new weight reservation, I believe, is waiting for a reservation on osd resources so that we don't have a whole ton of pg's doing that trimming at the same.

B

Time, uh weight scrub is a relation between this and scrub, where, if we're currently scrubbing the pg, we need to wait for the scrub to complete before we can continue.

E

um We are waiting on objects or for the.

B

Rpg, yes, I can't remember: let's go find.

B

B

It looks to me like we're waiting for scrub to.

B

B

Okay, I'm grabbing for it. It looks like you want to grab for a snap trimmer's scrub, complete.

D

B

It's called a pg.

B

B

And we call it scrub finish. So, yes, on script,.

B

A

So another question that gets you know brought up at the at least with the suff boost at various events, is, um is around rbd mirroring and, as I would like to understand, this is probably an after and after event thing. uh After all, these uh contacts are said and done that correct, rbd marion would just be pushing out to whatever other sites that would be available by snap context.

B

I am going to make some wild guesses that jason can correct if I'm wrong.

C

B

Snap mirroring works, is you rbd periodically takes snapshots and it mirrors the snapshots over. It then uses the rbd intent log to update the current version of the block image on those sites up to the current state of the current uh image. So it's it uses snapshots it's one of its components, but most of its rbd.

B

Level does that make make sense that is rbd itself maintains, I believe, a right ahead. Log of sorts for get some or mirroring purposes, and the remote cluster consumes that log to bring itself up to a point in time, consistent state uh time delayed by some amount. But it's not directly a function of snapshotting.

B

It's just one of the one of the sub components.

A

Yep that makes total sense just wanted clarification on that. It also looks like we had jason for a second, so, but it's not on right now,.

B

Oh well take that with a grain of salt, though I think that's correct, though it.

A

Sounds about right to me, I mean backward.

C

Is that snap mirroring is for disaster recovery, so it? I think it is just synchronizing the snapshots. Oh I see. Okay, it does them as frequently as like the pipe can maintain can can handle the updates but yeah. You just take that also in a temp log.

B

C

Well, there's also the the full data logging mirroring version.

B

Oh, I see I didn't realize that that's what that term specifically mentioned. That's.

C

The snap earring is a lower cost. One got it.

B

A

Greg and I guess, amongst all the different components that stuff has, are there any limitations in terms of one component to the other in terms of uh snapshot, features that you are aware of.

B

um Big yes, like cephas and snapshots, don't really resemble rbds right. It's built on the same substrate, but they're really different.

B

uh As as an example, a all of the objects within a snap within a an rbd block image will all have the same snap context, but that's not true of all of the objects in cfs um you, a a a a file in zfs can be contained in multiple snapshot domains. I believe they're called greg. That is, if you snapshot its directory, its parent directory in its grandparent directory all at different intervals.

B

Then the objects in that subdirectory will have the union of all of those snap ids.

B

From the point of view of the way self-managed snapshots actually work. This is irrelevant. It doesn't actually matter, but um the the way the feature is used to make zephyths's snapshot features wildly different for rvds.

B

Does that make sense?

B

D

B

Are integrated based on the way the component actually or based on the way the uh external interface actually works.

C

I think the more interesting one is when you move a file out across directories that have completely separate snapshots like patterns within them. They have a file that was in in in my directory getting snapshotted for like a month, and then I gave it to you and so now it's in slash home as just and it's getting your snapshots, but it's also still in my old set, but not my new set.

B

I forgot about that one, but the point is that, um yes, the features are different because the actual you know everything about zfs is quite different from rbd.

C

In in terms of snapshot, shortcomings, I think the big one is that these are read only there are some storage systems where snap, where you can take snapshots but they're, not but they're, but they're writable. So you can actually just have like different branches of versions of the file, and people ask us for that. One pretty often and we'd have to completely redo it.

B

Honestly, the only local file system, I'm aware of with that feature, is butterfest and it's not a well-liked feature.

C

Milders, do not have fathers, do it? Oh really yeah and I think a bunch of.

B

But that's not a feature differential between different interfaces and stuff itself.

C

And virtualized desktop environments are a big fan of it.

A

Yeah I was going to mention that as well. There have been a handful.

B

Actually, rbd does have that feature after a fashion.

B

Now that I think about it,.

B

They're not snapshots, it's not built on selfish. Well, it's not exclusively built on top of them, but you can create thin. What are they called? uh Sparsely provisioned copies.

C

For a paragraph you can you can clone a parent image.

B

There we go so rbd does have that feature after a fashion.

A

I think um another thing that we could talk about because uh you know one one of the moments for stuff was. uh You know, of course, during when openstack was first uh coming aboard and there was a need for shared storage um having a way to give storage to different virtual machines at the beginning. um Maybe you could kind of talk about the cloning and snapshot on that with rpd as an interest point.

B

So say: basic concept: I'm gonna, I'm gonna tell some high level, probably truths, but probably not strictly accurate.

B

um So at a super high level, an rbd image is a set of rados objects, the simplest version and I believe the default configuration is where um you simply divide up the image into four megabyte chunks and each trunk has its own radius object.

B

So a really simple way to create.

B

Let's say you have the sort of classic example of you want to create a base os image, let's say for rel whatever, and you want customers of your cloud deployment to be able to quickly spin up new vms based on this uh relavage.

B

So you take a snapshot of the rel image and then you create a new block image with that rel snapshot as a base, and what that means is that rbd maintains a bitmap of all of the blocks that actually exist within itself and anything that is not in that bitmap. It passes through to the parent image, in this case the clone. That was the original rel image and every time you do a write. It copies up the and, I believe, that's the operation of the liberty source code.

B

uh It copies up the parent images for megabyte block up into the child images uh layout and and updates the bitmap to ensure that all future reads will hit that block instead of the parent one. In that way, you get this kind of thinly provisioned behavior, where you don't actually create a new image.

B

You just create new blocks, corresponding to the things that have been mutated since, uh since you created the the image, um so just as with the mirroring thing it makes use of snapshots, but most of the behavior is actually rbd level.

B

That I am not familiar enough with to find the source code too, though,.

A

That seems about right from how I try to explain it to people as well, so.

B

And I'm I'm pretty sure it's a bit map. I think it's pretty just basically maintained on the head object.

A

Great well, I wanted to also check to see if anybody else wanted to speak up with questions as we are approaching the end of the time for the code walkthrough.

A

Okay, uh sam was there any last things that you wanted to share in terms of uh oreno's snapshots.

B

uh No, I think that's all I have. I hope that was useful.

A

All right, a great thing, thank you, everybody for joining us for another stuff code, walkthrough and, as I mentioned before, this will be posted up on the stuff youtube channel. While I have everybody's attention as well, I wanted to make a note that the stuff really shirts for octopus have finally have gone out. The design for pacific is on the way everybody's hard work on the next pacific release for this month and lastly, we have the stuff user survey. That's ending april 2nd. So please fill out that survey.

A

When you can, you can put it into chat real fast, but otherwise we have another tech, talk that is coming up and it will be in on the 25th.

A

So this coming this week and we'll be having a specifically a presentation on bucket persistent notifications, so invite everybody to join us for that. That's at 1700, utc and there'll be updates on this f mailing list and twitter as well. So, thank you, sam again for taking the time to go ahead and walk through with us on radio snapshots and answer our questions and uh we'll catch you all next time for the next f code. Walkthrough next month,.

E

C

A

Yeah have a good one. Everyone bye.