Ceph Ceph Code Walkthrough, 10 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ceph Code Walkthroughs: Librbd Part 2

Description

Every month the Ceph Developer Community meets to discuss one aspect of Ceph code, spread knowledge of how it works and why it works that way.

This month we're joined by Jason Dillaman to go over part 2 of librbd.

Part 1: https://www.youtube.com/watch?v=L0x61HpREy4&list=PLrBUGiINAakN87iSX3gXOXSU3EB8Y1JLd&index=2

Find future Ceph Code Walkthroughs: https://tracker.ceph.com/projects/ceph/wiki/Code_Walkthroughs

Ceph Code Walkthrough Playlist: https://www.youtube.com/watch?v=nVjYVmqNClM&list=PLrBUGiINAakN87iSX3gXOXSU3EB8Y1JLd

A

Welcome everybody to another episode of theft, code walkthroughs. It is a february 23rd and I am joined by jason here, who is going to be giving us a part two on live rbd, as we have uh previously have uh gone over in part one. So uh I know that we've had a variety of people that reached out for part two and so jason is giving us some time now to go ahead and go over that so jason. Will you please take it away.

B

All righty, thank you. um So, yes, in part, one uh we covered just the introduction to how io's uh read and write requests are uh sent to lib rbd, um so I'll quickly. Recap it and then we'll dive a little bit deeper, uh touching on some of the more uh complicated uh state machines. Then, in the in the I o path for this one. um Since I cannot see anyone raising their hands because my my screen is now taken up with a terminal window uh feel free.

B

If anyone has any questions, uh comments or whatever um I'll meet yourself and uh make it known. um Otherwise, I will not know. um I.

A

Will not see the chat I'll, try my I'll try my best here to monitor chat for you. So.

B

That also works. Thank you all right, um so yeah at the end of the last time, or during part one um we talked about where, um basically, how ios enter uh read, write, request, enter, live rbd um and in general we talked about how this, um if we look at where we are in the stuff tree um under source live rvd is where we keep um nine 99.99 of lib rbd related uh functionality um and there's one uh cc file in here called lib rbd.cc.

B

This is basically the entry point for the c and c plus plus apis for the lib rbd. So everything in this file um when it comes to like read and write methods, um they don't really do a lot in this file, but it redirects somewhere else deeper within uh librbd. So this file is basically what keeps the uh the stable api that's available um in the library.

B

Apis so like in this case, this is a c api. So every function you see in the in the library.h or library you see live rdd.http you'll find uh the corresponding implementation in the library function. um For the I o path. uh The function then gets basically bounced to. um uh If you look at the subdirectory layout here, there's a there's, an api subdirectory.

B

So that's where we start breaking things out into different, uh lower level components and features within liberbd. um So for io, all the idle method starts within this api. Slash, io cc file and then you'll find that the corresponding, read and write methods that really all they do is um yet another layer of indirection. They kick off uh in this case. What's called an image dispatch spec an image dispatch spec just describes an I o um generally um and uh gives it any details like.

B

If this is the right operation, we have offsets and links that we're going to write. We have the the data we're going to write any flags that we're going to send um different different ios, create different dispatch specs. So that's, essentially all right discard creates a discount uh dispatch back a right same.

B

That's when you get it just like a little bit of data, and then you tell it to keep writing the same data over and over again uh different dispatch, specs for that, so on and so forth, um but the dispatch itself, just that stuck itself, is uh what takes you really into the I o subsystem of lib rbd, which is uh essentially most of it, is under the live rbgs flash io uh subdirectory um in the uh we talked about the dispatch back.

B

One thing you might have seen there was that uh it has this concept of uh image dispatch layers. It starts off at the api start, but as the I o moves through uh the dispatch dispatching layers, we have these plugable layers that allow different functionalities.

B

So the first one is just to cue the event so that we can return control faster to the the caller application. We have quality service, throttling exclusive lock uh blocking. We don't allow any any rights to proceed. If you don't mix with the block uh refresh date machine, that's if the image has been modified somewhere and you need to refresh like snapshots or something like that from on disk, we'll pause the ios. While we do that refresh, we have a migration layer that handles any ios.

B

While an image is under migration, a journaling layer doesn't do anything, I think right now, but it will, in the future a right black layer that we can just use internally to basically pause all rights to the image. Let's, if we're going to take a snapshot, that's how we can quiesce any in-flight ios. Before we proceed with the snapshot, we have a cache layer. uh We have brand new with pacific.

B

We have a crypto layer that implements lux1 and looks to uh encryption uh directly within librbd and then, finally, last but not least, we have the core layer, which is actually the meat of it, um as talked about in part one and that's the part that then takes a given image. Extent based. I o is when we talk about image extents when you think of the block device- and you have you- know- one gigabyte block device when you reference any section of that of that block device.

B

Let's say bytes 0 through 1k, that is in uh image extent, so you're referencing image, extent, byte 0 through image extent, byte 1024.

B

What the core layer will do here is then translates that image extent into object based extents, because rbd itself is a given rbd block device is made up of uh you know by default, four megabytes docking objects um that are striped across the the object size. So if you have you know a 32 or, let's say 16 megabyte image now you have uh you know. Four different objects object, zero, one two and three that each represent um four megabytes of that extent.

A

So and jason, can you remind us what the slayer refers to that we're looking at here yeah this.

B

Is the plugable layer for how I o is handled? um So, as I've said up here, the we have these different plugable layers that just take an I o and they can do different things to it. um Like the q, one will and cue it like. Just basically put it in a queue to uh uh a thread, will take it off pop it off the queue in the background, but allows uh us to return control immediately to the api caller.

B

While live rbd continues on its way quality services, throttling exclusive lock is making sure that you own the exclusive lock before the I o can proceed refresh um so on and so forth. These are. These are all different things we'll we can dive into we'll dive into the um the object dispatch core layer uh again, um but these were all covered at a higher extent um or in a deeper extent. In a part, one.

C

So are these layers in the uh like uh communication protocol layers means no.

B

They're so literally they're, just at different class instances, so um here's the with qos cos image dispatch. So this um this class instance we will, when the image is open, we'll create an instance of this class. Then what happens? Is the ios get basically passed from layer to layer to layer to layer in that order that you saw in the enum?

B

So in this case, um if this layer could get a read request, that read request was described by the image dispatch, spec and it could say like well, you know do we do we have in this case it's a new feature to exclude any particular operations from qos, but otherwise, if you, if we need to throttle this operation for a read request, um you know this needs throttle method actually. Does it and then basically stops the further dispatch of that I o until eventually that um until eventually, basically, what happens?

B

Is that the uh you know when the throttle becomes ready, then we can basically release the I o again and let it proceed to the next layer um down in in the image stack. So this is just the original version of uh you know. From several years ago, when I took over lib rbd, it was you know the equivalent of a bunch of if-then-else statements about uh running through I o flow um and then, as we added more and more features to lib rbd, it became uh an unmaintainable map.

B

So this just allows us a way to basically segregate logically um different sections of code. um You know the example with the um the new pacific feature, which is crypto um we now we have the new image dispatch.

B

B

So yeah the the crypto, you know it handles you know. If we, if we're going to encrypt the disk uh decrypt, you know this layer can get dynamically loaded. Only if encryption is is enabled, then it can handle. um You know uh decrypting and encrypting decrypting, reads and uh encrypting rights uh on the disk transparently, and it was it's a layer that you know we can dynamically plug in and we don't need to basically add little if that else hooks throughout live rbd.

B

When we wanted to add this function, we can tightly constrain and keep all this logic uh consolidated to you know one specific spot. Does that answer your question.

C

Yeah well maybe uh the next question is layer to layer. Is there a common interface between a layer to the layer above or below it.

B

Yeah so there's uh there's a templated uh dispatcher.

B

And so this this little helper. um Let's say this gets there's two versions of this: there's the image dispatcher and there's the object dispatcher, because they we have image layers and we have object-based layers based on those extents, the same functionality, but basically just it. It finds a given dispatch layer from where the the I o first enters the system or where it next needs to be handled, grabs the dispatcher for that layer um sends the op to the dispatch layer um and then, if the dispatch layer says it's it's handled it, it returns.

B

True um we're done um so, that's basically how we can pause the the further approximately otherwise, if it didn't handle it, um it'll just keep going to the next layer to the next layer to the next layer to the next layer. Until it's it's hit every single layer in the in the stack.

B

um So when you, when you look at um the image dispatcher.

B

When it comes to actually sending it, we just use a visitor pattern and um that can basically take the different image in this case. It's an image dispatcher, so it takes that image. Dispatcher back and the very spec type it is via to read a discard.

B

All right, write same, compare and write flush whatever, um and it invokes the associated method um on the provided uh dispatch layers pointer. So that's how based on the the incoming dispatch spec, we can then invoke a specific function within a given layer uh generically um without having to hard code all this logic. um So, but when you see the logic here for image dispatch, it actually, then.

B

Object, dispatcher has, you know its own version of the visitor, um but it based on the object of sketch spec, which is similar in concept to the image dispatch spec. But it's based on object's extent and an object extent is uh basically byte ranges within a given object. So a given object, number and then byte ranges within that object, because when we map, when you think about the standard, rbd image layout of a four megabyte object, um if I tried to access, I o four megabytes through uh uh through, uh let's say five megabytes.

B

The uh image uh dispatch layer will process it and eventually, when it gets down to that core layer, um it will convert it into individual object, dispatch spec for that request and the object dispatch specs. In that case. In the example I use the four megabytes to five megabytes that object, dispatch spec would be object, number one um and then object expense, zero, bytes through one megabyte, because that's how that striking pattern worked to to map that and that striping is handled.

B

In the in this state machine where this is this is the core layer. This is pretty much uh more or less legacy code that uh it has been around. But what we do. We have a generic. um In the case of a generic write request, we basically break the request down into the individual object extents and then for each of the object extents, we can send.

B

um uh Basically we can create an object request on it um and then based on the type, because this is a generic base class of abstract, but if it was derived from the let's say, a write request. The create object request here just goes and creates a brand new object.

B

Dispatch spec, representing the uh object, number object, extent offset um and the object extent length which we can basically interpret basically the length of it from the length of the buffer list packed in because we would already snip the buffer list from the image extent point of view to the object standpoint of view by this point in time.

B

um But yeah getting back to the dispatch layer uh where we kind of left off last time uh was in the object, dispatch layer. um We have the cache layer, which is basically a uh that's here right through right back right. Around uh cache layers, uh we have the new crypto layer, that's actually where the the meat of the luxe 1 or lux 2 based encryption is, uh is handled in lib rbd, it's handled on, object, low level objects.

B

We have a the journal layer. This one basically makes sure that rights cannot proceed until it's been. The journal has been properly flushed to disk. That way, we don't end up inconsistent between what's in the journal and what's actually on the disk, we need to make sure the journal is up to date and committed to this before we let the I o proceed.

B

We have this new thing from octopus called the parents cache, and that is something that allows uh read, requests to be uh cached locally for immutable parent images. So if you had a a golden image, let's say of an operating system, then you clone that image to a new block device.

B

Any reads from that: black device would normally get serviced by the parent image, but to offload that we can apparent, cache can write those persistently locally to the hypervisor host in order to satisfy future reads.

B

um Next, one is the scheduler that all the scheduler does. It does a very simple disk, scheduler to basically try to batch up operations or that are happening against the same same object. So if you had a bunch of small sequential reads or a bunch of small sequential writes that are that are occurring at the same time, it'll batch those up into a single operation to the osce, so that it doesn't overload the osd with you know.

B

Hundreds of requests with a single request could have been enough and then finally gets down to again the core layer and that's the meat of where we then start sending data to the to the osds.

B

And in this case the core layer corresponds to the object, request, state machine.

B

Similar concepts for given object request week, you know we have rights, discards, right chains, comparing rights and reads the way it's broken down. uh We have the the object, requires the base class and then we drive from that um uh classes like for the specific operations like in this case. Here's a read request.

B

We again try to document. You know through little pictures about how the state machine progresses. um In the case of a read, we first go to the osd and try to read from that associated object, but for the osu tells us that that object does not exist and it's a cloned image.

B

Then we could then reissue the read request against the parent image to try to read that data and we'll dive deeper into that. um Then, in terms of write requests. They all derive from this abstract object, write request, um but then you have a little more complicated state machine.

B

If you have object map enabled we can basically detect that it's there's nothing for us to do. Let's say, there's a discard and we know the object doesn't exist. We don't need to send anything to the ocs. We can just know off the right and uh and continue, then, if we're uh actually writing, we might have to pre-update the object map before we actually write to it.

B

So if we're actually going to write data and the object map says that the object does not exist, um we would need to update the object map first before allowing it to proceed, um and then I mean there's some more optimization paths about like well. If the object mike told us that it doesn't exist, then we can go straight into the copy of the state machine and we'll get into copy after that has to do with. How do we take data from the parents and and copy it over into the child?

B

um Otherwise, if we know the object exists, uh we can just actually issue the right against it and then, finally, the last step is the post update, object, map stage that actually only gets hit uh if you're deleting the object, because it's when you're deleting the object, the update state is, it says it's a pending mode, that's pending removal and then the post state actually flips it through that the object no longer exists with a guard on it to make sure that if you had other, I os coming in that hit the same object that it won't.

B

uh It won't affect people and won't stop on each other. To say the object doesn't exist if it got recreated. In the meantime,.

B

So yeah, if we um dive into this path, a little bit um start with the the read.

B

Yeah in general, any any object that uh any object that comes in or any request that comes in. uh We have some helper methods at the top um in terms of right hints that for writing anything we can. We just tell the osd that you know it gives us a max object size. um It was important file store.

B

I don't think it's really that important with blue story or more at least it doesn't seem to be, but they're legacy, wise, uh compute parent expense, helper method um when we talked about, if you have a child clone child image that hasn't been overwritten yet and we ever need to get the the parents data from it.

B

This basically helps us know that when we, when we take the object extents of this uh child object, we basically transform that coordinate space back into the parent image, because the parent image coordinate space might actually be different than the child images chord in space, um and you can think about that. As an example would be that uh if I again, the child had a four megabytes striping size by default, and I was reading starting at four megabyte offset.

B

That corresponds to object one uh in the child because object zero covers by zero through four megabytes and then object. One covers four megabytes to eight megabytes, um so object one, the the offset of zero, because that's the start of that object of where the image extent of four megabytes starts.

B

But if I needed to read from the parent and the parent had a an object size of eight megabytes instead of the default of four megabytes, um when I read from the parent I actually would not be reading from object one I would actually be reading from object, zero and my objective that would not be offset zero it'd, be object, offset four megabytes because the in the parent, if you have eight megabyte backing objects, uh the first object object. Zero will cover the space of the object, extent space of zero megabytes through eight megabytes.

B

So, if I'm reading uh for starting at four megabytes, I would read from object one. So this is just some generic math to basically do all that transformation um back into image. Extent of the of appearance, um so yeah uh getting into the into the read state machine.

B

um First first entry point here is just as the read object as as the diagram kind of showed that you know if, if we have this little optimization here that says, if we're reading from the same snapshots of uh where the image is currently set to, and we actually have an object map object is not no pointer and the object map says that there's no way that this object exists, the object may exist is is false.

B

Then, basically, we can skip this step and go on to the next step, which is the the read from parents of the of the I o state machine, but otherwise, if we can't, um what we do is for all the object. Extents that are present in our system are in that in that request, because we might want to, we might do a request. That's you know, read bytes 0 through 1k and then read 5k to 6k. We support these. um You know io vector style, reads and writes internally within within live rbd.

B

um We kind of split it up and say. Well, you know we have this little helpful hand such as you know, if it's greater than this uh threshold size, which I think is set to 64 kilobytes by default, then we'll tell the osd gif give us a sparse read. um So the osd won't retrade a bunch of zeros. If there's no data there, they'll just tell us that hey there's, no data there from this extent. To that extent, otherwise just do a standard generic read and then uh for a standard. Read the osds will say.

B

Well, if there's no data in that in that space you gave us, um but the object does exist, we're just gonna. It's gonna get zero filled um for for that length. You told us um that doesn't exist and then it just sends that request um uh to the osd and then when, when the osd comes back, it's going to call the the handle method. So in this case the handle read object uh so yeah. If the. If the object itself did not exist, this is not about. If there was data there or not.

B

If the object itself does not exist, the oh, it will return. uh E no ends error to us and then we'll basically go into the the read parent state machine. If we get any other errors, we'll bubble the I o air up to the to the caller.

B

Otherwise, if if the osd was able to read some data, we're done because what's going to happen is is that on that read operation we um we gave the osd a space to.

B

um We told it where to put the data when it when it reads it, so we gave it a place to say here's, the buffer list of when you read that data put in the software store for doing a sparse, read, we said, put the data in this buffer list and also give us the sparse extent, mapping of what data is actually in that buffer list and how it maps to the object, object we'll dive deep into a dive deeper into how that then gets converted back up into image space, because this again is object.

B

Space so extend space. uh Those buffer lists then need to get reassembled back up um potentially into image space, because that that's what the caller uses, they always reference. The ios in image coordinate space. um So, if I was reading by zero through eight megabytes, that's two different requests, but for a four megabyte default shaping size. So that's gonna get broken up into an object request against object, zero and object requests against object, one. We send both those requests um at the same time.

B

uh You know in parallel to the osds I mean they're, potentially most likely on different osgs and different pgs, we'll get the responses back uh at different times um so, but this class in itself just represents one of those requests and the next layer up actually handles them. Reassembling and basically concatenating those two separate buffer lists in that example, back together into a single buffer, lift um that will be used uh and expected by the by the client.

B

um So yeah the in finish here, if the data, if it was able to read any data, um otherwise it'll, go into the next step, which would be try to read from the parent. uh Assuming we have some internal flags to basically say: don't don't read from the parent, but assuming that's not turned on um yeah, then we basically kick off a helper function. uh This is uh semi new, I think with uh pacific, uh but really all it does.

B

Is it takes that um those extents and does that conversion from this this to cloned or child images, object space and converts it back into the parent's image extent, space and issues a read in the exact same fashion that we saw before through the uh to the api. I o layer to kick off a brand new image extent read, but it kicks it off against the next image in the in the chain, which is the parent image.

B

So in this case this is essentially acting like a generic lib, rbd client like kimu, because it's it's issuing a read, request, an image, expense space and it gets a response back in image. Extent. Space and that's uh uh you know just handles here. Then so, eventually the data will come back if if there was a parent um it'll get handled uh it'll fall through. But if there's no data.

C

It'll get an e no end so.

B

That's by the time you think about it as a recursive algorithm, where it's you know following the chain up and causing a read on the next parent, the next parent, the next parent and the next parent up. That's how we can support. um You know clones of clones of clones of clones of clones. It just keeps recursively calling down into um each parent and issues reads until it actually gets some data.

B

um So if it gets back to here and there's no data, we know that no parent has any data that covers that extent, and we can just uh handle that generically. When we reassemble the I o uh we can still. We can basically give a buffer of zeros back to the user and say yep, there's no data there, not that you didn't care, but it was all zeroed or if there's any errors, we can bubble that up. um The last step of the state machine is copy up, which is essentially on the read path.

B

This is basically a copy up on read uh it's not enabled by default um we're gonna we're gonna get into the copy update machine. uh Next, after we go to the write state machine, um so I won't dive into this too much.

B

But basically it says like if, if this feature is enabled, if we're as long as we are doing a copy up on, read like and that's something that someone said they wanted to do and opted into this, then we'll we'll kick off a copy up request to copy the data from the parents um and copy it and clone it down into the into the child image so that we never have to read from the parent again. For that given object, extent.

B

uh The right state machine uh very similar, but in terms of where the state machine starts uh in the sun method, um we kind of have the the same. Optimization here you know. Well, if we don't have, if we say we don't have an optic map, you know, then we just have to assume the object may exist because we don't know without the object map.

B

Otherwise we can look it up directly in the object map to ask it if the object may exist, because if it knows for sure that the object does not exist, we can potentially take some take some optimizations here like if we were trying to remove an object and the object map says it doesn't exist. Then that's the example of how we can skip a null up on a on a non-existent object. We can basically end the state machine early without a round trip to the server.

B

You know if someone was issuing there's a brand new rbdm engine. You did a makeup s xfs, which loves to do a giant like whole disc discard, and you had object. Map enabled, um if you didn't have object map enabled uh you would just literally have to send discards for every extent of the image which you you know, the osb would say: yeah, there's no data, there's something for me to remove, but if you had object, map enabled and the object map is basically saying. No, this data, there's no data here.

B

Nothing exists um that makeup makeup s call is basically no opt away into a a very fast operation instead of a very long operation. On long images on big images.

A

Hey jason, um while you're in this layer, can you also uh at some point cover how a delete recall sorry, delete requests uh happens in the flow as well.

B

Yeah so delete is a is a write. Request of this abstract object, flight request. It covers rights, discards which are uh discards that are zeroing extent within an object. It's truncating the end of an object or it's removing an object entirely.

B

It handles the copy and write it and it handles right same operations with via the derived classes. So well, the the general state machine is the same and we can dive into the the the lower level augmentation.

B

So um as shown on that little ascii diagram, uh you know now we're here. The next step would be to uh update the object map to make sure that it actually says that you know, for if we're actually going to write some data, we want the object map to say that the data that the object exists or, if we're going to remove an object that exists, um we need to first set it to the pending state.

B

So if the object map has been disabled, we will skip out of this state and go on to the next one.

B

We have an optimization here that says again that you know we know the object doesn't exist, so we're just going to invoke the copy up state machine directly, because the copy upstate machine will also handle updating the object map. So we don't need to handle this here.

B

Otherwise, if it's just a plain, vanilla, write operation, all we do is we can tell the the object, map and say: hey asynchronously, go update the object map on disk. For this given object. Number with my new state, the new state is determined by the drive class. These are virtual functions and then, once that completes, uh we get ins. We advance into the handle.

B

uh Pre-Write object, map, update state which doesn't really do much, except for check for errors, so it can bubble the error up, um but otherwise it proceeds to the actual write phase of the write request and again we're talking generic it generically. Here the write request is any right: operation, discards, rights, right fans, uh zeroing, truncation, removal, uh compare right, yada yada, um so the first step here is um if we potentially have copy up enabled and the definition of if we have copy up, enabled there's two cases here.

B

One is if we're live migrating an image. We need to basically ensure that two different clients can't stomp on each other.

B

So we we have this special assert here that when the when the I o gets to the osd, the osd will evaluate this assert. That basically says like hey. We are asserting that the current object. That's on disk, has a particular snapshot. Sequence number, if it doesn't error out and we can detect that error so that we know that we're doing like a scale, a scale write operation or something like that to avoid stomping on someone else's changes.

B

So this this is new logic. When we added uh the live migration feature in um mimic or no, I can't when it came, but otherwise the the normal path is.

B

The original path was just this uh assert that the object exists, and this is for the case only when you have a cloned child image, because the very first right to a child image, we need to make sure that all the data from that overlapping object extent is copied from the parent image down to the child image and then with your uh right applied on top of it.

B

So if the object doesn't already exist um and that's what this is sort of exists, check uh handles, um it's gonna fail out and not actually perform the right operation and it's gonna. Allow us to basically go sideways um in the state machine to go and do a copy up request, which is a different class.

B

So yeah move forward. We add right hints those were those things at the top of the file. That just say like the object is going to be. You know four megabytes in size. We add the right operations. These are each derived class will inject their own write operation.

B

So this is where, like a a delete when put in a sleep right or um a uh or a compare right would put in the compare operation and then put in you know the the compare and write operator and then it sends the request to the osd for the current context for the currents we convert, that object number into the corresponding rbd underscore data.imageid.object number, and when we get the callback from the osd asynchronously we'll advance the state machine to the to the handle right object state.

B

um So yeah the what could happen here. The the first error condition we could get would be the e no end, and that really can only happen because of the assert exists failed. So the assert is exists that the object doesn't exist, so it fails out and aborts um atomically aborts the right operation uh saying uh with your code enough.

B

So that's how we know to go kick off the copy upstate machine if we got a e range and we regarding uh guarding that right operation because we're doing a live migration, um that's how we knew. I know again that oh we gotta go. You know through this special path to go, do a copy up and the copy upstate machine also handles this live migration path.

B

Otherwise you know there's this race condition case where you were flattening an image while right into an image, so the parent might actually disappear while you're uh an il is in flight. So this is just something that says: oh uh you know. Let me let me go restart the state machine back up at the top um uh illegal sequence. That's from the compare and uh and write operation.

B

That's the theore code that comes back from that one and then any follow-up, the generic errors will will just bubble back up to the user and then finally, we handled the post write object, map update case, which, as I mentioned, this really only matters for the remove operation because it flips the the pending state to the non-existent states for the given object number.

B

So this right here is basically a filter that gets atomically applied or on the osd to say like what the current object map state is pending and only if it's pending change it to the non-existent state and then finish off the I o operation.

B

The one thing I skipped over here is the is the copy up and that's going to be the next day machine I'll dive into. But what it does is the first thing it does is. It looks um to see if we already have a copy up in slice for a given object number, because I could have multiple ios that all land against the same object that might kick off multiple copy up state machines. Concurrently, we can actually then coalesce those actual writes down into the same copy up.

B

um So if we couldn't find it we'll create a brand new copy, update machine keep track of our our new requests copy up request so that any other request can find it and then start that state machine off, and otherwise, if we already found an exact match for an inflight copy up for this given object, number then we can just append ourselves um to that request, um and what's going to happen is is that when that copy, upstate machine runs and it's finally getting to the point where it's going to then add in all the right operations, it'll just go in order that it solves these rights appended and append those light operations to its to a single request uh to the osd.

B

So yeah we get down here, we can start seeing all the drive classes, so an object, write requests, um you know for its right hand versus write-offs, the right ops it does it literally um it's either gonna. If you're, if your write request was gonna, hit up the entire extent of the object, it just calls a slightly different osd operation, which is rightful otherwise. It just calls a write with an offset and the data you wanted to send this a discard.

B

This is where it starts getting. It's got different actions. So if we know that we were delete discarding and we were gonna result in the entire object getting uh discarded, we can actually remove the object, um except for one case, which is the fact that if we were a cloned child image, we don't when we do a discard, we still want to hide the parent image. So what we actually do is we create a an empty image and truncate it down to essentially a size zero.

B

In this case, the the truncate case is if we were um discarding- and it just touched, the trailing end of of a of an object like. So if you had a four megabyte object and you wanted to discard- and it hit object except two megabytes before megabytes, it would just truncate that object down to two megabytes inside um and the zero um it's just. Whenever you have a discard, that's going to hit somewhere in the middle, not at the end, and it doesn't, it doesn't take up the entire uh object itself.

B

uh It'll, zero off specific uh extents within the object.

B

uh The compare right same uh it has a right same operation, just copies the data down the compare and right it uh invokes two operations. It does this compare extent which is kind of like a guard. That's the guard operation to say, make sure the data at this object matches this buffer list, and if that atomically passes then atomically. You know, then do that the actual right operation, be it a rightful if it's a full object or the right, um and it's got this special filter.

B

The right result case back just because we need to store the the mismatch offset of where the the data first mismatched in the I o and the osd kind of gives us that data through the um the return code. So we just have to extract it out to get the object offset. So we can return it back to the user and convert that object, offset into from object extent, space to image extent workspace.

B

uh So that's what this is actually doing, because the user, when they want to know the where it mismatched they want to know image, wise or mismatched, not object, extend but based uh where it mismatched.

B

uh And last but not least, this is uh new with octopus, but this is a state machine that just handles uh listing snapshot dips, um it's a little complicated, um but I'd rather get into the copy update machine.

B

If there's no immediate questions.

A

I know you're good on chat right now, so.

B

Okay, so the copy up estate machine, it's again in the I o subdirectory, it's the copy up request.

B

In general, um well from the little ascii chart here, it's uh how the state machine functions when it when it first starts up the the first thing it does. Is it um if we're doing a standard, a standard clone child image having to copy data from its parent image down to the child image, so that we can actually overwrite some data?

B

The first step it does. Is it reads that corresponding data from the parent image and then proceeds if we're doing a live migration? We take this alternate path, which is a deep copy path, which is a different state. Machine um probably won't have time to get into because that one's a real, a real doozy, but what the deep copy does um it actually handles uh going through and reading snapshot deltas from a parent.

B

So if, if I have a parent image, that's got snapshot a b and c and those snapshots a b and c have have different data. For you know those different uh snapshots.

B

The deep copy will actually just take the deltas for each of those snapshots from each of the parents um and then reapply just those deltas to the to the child image. And that's: how uh live migration handles the um the process of of getting a a fully deep copied, but yet still sparsely allocated uh version of your source image to your destination image in a consistent fashion that doesn't uh require exclusive locks or anything like that, but yeah. Once once we do the the reason parent.

B

We can update the object maps and it's mass plural, because we might have uh one or more snatch uh one or more uh snapshots on our child image or uh destination image in the case of live migration. um So we would need to update all object maps from beginning of time to the current head revision and then we actually perform the copy up operation.

B

So um state machines- maybe so I can see a pattern here. Entry point is defend method. um The first step, though in the state machine, is just reading from the parent.

B

A little weird case here that, if there's a potential race condition between kicking off a copy up request and the um the parent image getting flattened away and removed from the child image, um we can just return that saying: hey the parent image doesn't exist anymore, um which would restart the state machine uh back in the object request state machine um if there's a deep copy, it get live migration. It goes to that alternate path from that uh ascii diagram, like I said, we'll, probably have time to cover that here.

B

Otherwise it does the the normal quote-unquote generic uh live, rbd or rbd copy up operation. um The what happens is, as I mentioned before, to read from the parent um it at first and it had gotten the apparent image extents um or just the image extents that correspond to the child's object that was being copied up from the object request state machine. So it's part of the constructor for um the copy up request. If it passed the image extent of what what this current object represents. Image extent was um so what happens?

B

Is the copy state machine it just kicks it? It creates a brand new image to spec image dispatch. Spec read request against not the image itself, but the parent image, so this parent is basically a pointer to another image context. So that's how you can start getting the chain of chain of a chain of a parent parent up the layer.

B

We start. We we tell it that we're going to start dispatching it at the start of the internal layer, so that bypasses like those qos layers and bypasses the um the queue layer. The other things that normally are only meant for the end user api, uh give it the completion function, which is what's going to get called when the date is available.

B

We give it the image, extents that we're going to read from we we create a new read result. The read result basically is a way for us to track where to put the data for the read: that's the mapping helper. In this case we do a sparse read, so we give it a map and a buffer list.

B

So if the, if the parent actually had no data or if I'm requesting to read for megabytes and only had one megabyte of data available, you know the the buffer list here would only have one megabyte and the map would tell me how that one megabyte maps back um to the to the image and then kick it off.

B

uh Once we get the data back from a parent uh assuming it's not a uh a failure of some, I o that we need to bubble up uh we. uh uh This is. uh This is a new thing related to pick crypto, because if we have like a lux header on the um on the image, we just need a generic way to basically map uh the data and basically slide the data down or up to cover up the lux header.

B

um Otherwise, um so now that we've gotten the data from the parents, uh if you remember that uh copy up request list, we basically take ourselves off that request list, because now we're getting to the point where we're going to actually send the right requests and we don't, we don't want any incoming ios to try to append itself to us. While we've, you know already sent the data to the osd um good.

B

The next step is we prepare the copy of data and really all that does. Is it it kind of like loops through all the requests. The object requests, instances um that were tagged and appended to this copy of request and say uh give me your data. um You know if you're a write, request or you're. You know give me your write off and things like that. um So now, there's some optimizations here to know you know if everything we're writing is gonna be zeroed.

B

If we still need the copy up, we can dive into those. But these are just optimizations. They they're kind of tangential to the actual logic itself um and.

A

Well, jason: we are approaching towards the end uh for the session. So would there be anything like in summary of what we have covered today, that you would like to leave the audience with.

B

Yeah, there's never enough time. It's number one um yeah, I'm trying to get through the the copy update machine, um but that's in general the object flow. The only thing that was left was when we were talking about um reassembly of of data, and maybe this is just an exercise to the reader.

B

But that's um when we talked about the the read results, um we can basically use this helper to reassemble the data back into a form that the caller was expecting it in again in a generic fashion, because we have different formats that we want the data end up in. We have different source formats, so the data could be specified and it could be sparse or it could not be sparse. So this kind of is a generic thing that covers all those cases.

B

um You know for reading image, extents versus object, extents, uh so on and so forth, um so yeah uh in general. That was the the I o paths and how it bubbles, through all the layers and gets to the to the to the osd's and back um any high-level questions in the last five minutes, or so.

A

I'm looking at chat right now we don't have any particular questions. Listed, you've already covered uh some of them and well you covered all of them. Rather so, let's see uh if anybody that is on blue jeans wants to speak up now would be a good time. We got about five minutes left otherwise I'll be watching chat.

B

Okay, um so in general, though we, the important thing to know is uh that we try to let you follow the state of the world through at least pictures to to try to make it easy to follow the control flow um if you're, just looking at this from the from the outside, and just knowing that. Basically, everything in lib rbd is a is an asynchronous state machine where we send a re, kick off an asynchronous request via deliberative api and eventually we're going to get a call back um to our handler function.

B

For that request, and that's how we advance the the state machine, I think that's in general, like how you could approach uh lib rbd, every state machine is its own, is its own uh file. Usually um just uh at least everything new is in its own file. um Just try and make things easier, uh uh given the kind given the potential for complications be uh make things easier to to understand and follow the flow of.

B

And, like I said, everything is 99 of it is, is in the library subdirectory and that's where your key views in the world are gonna are gonna attach to and utilize. um Even uh if we're gonna look at other functions, we have things in the tools directory. We have the the rbdclis and tools um for rbd, mirroring that's a subject in itself, but yeah.

B

Obviously the rbd meteor daemon is: is there um the rbd network block device uh demon is got its own directory and so on and so forth, and this is this is newest pacific. This is uh the new windows uh mbd uh demon from suze and cloudbase, uh but it's yeah. um We try to keep semi organized where everything goes um to make it hopefully easier to understand. But if there's questions comments or concerns, I mean there's the mailing list. There's irc um feel free to reach out.

A

Okay, well, thank you, jason for taking the time and covering with part two of rbd. It looks like we still have even more to cover, but uh I, of course, don't want to take up more of your time. uh While you know we're working on the pacific release because, uh hopefully we're looking at early march right.

B

I think that's your goal. Yep.

A

That's always the goal, so um thanks everybody for joining us live and, uh of course this will be recorded and put up on the archive uh and on the ceph youtube channel. So thank you jason again for taking the time and uh we'll see everybody on the next code.

B

B