Ceph CephFS Code Walkthrough, 27 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CephFS Code Walkthrough: CephFS Mirroring Part 2

Description

Presented by: Venky Shankar

Schedule: https://tracker.ceph.com/projects/ceph/wiki/CephFS_Code_Walkthroughs

A

Again, yeah hello and welcome to this uh talk, follow-up talk for demurrer daemon, so the letter we saw mostly the interfaces and uh how things tie up into the uh manager module the mirroring module uh and the inner workings of the mirroring module uh yeah. So this talk is about uh the daemon itself, uh the daemon, which actually does the synchronization of snapshots between the primary and the secondary cluster file systems in the primary and the secondary cluster, so yeah so the uh so.

A

This ffs memory, daemon, is uh one per cluster, uh which means uh the merit demon can handle uh multiple file systems uh since fms is multi-fs ready, so it made sense to handle multiple file systems, uh so you need just one mirror daemon for the entire shift cluster and the mirror daemon needs to be run on the primary cluster.

A

uh So uh this and is based on a push model. So uh the data is pushed from the primary cluster. The snap data is pushed from this primary cluster up to the secondary cluster file system in the secondary cluster and the mirror daemon uses lips ffs. So it's a normal client application.

A

um It uses lips ffs to talk to both local and the remote cluster uh while designing this, we are thought about. uh You know doing mounts in the primary on the second and the secondary cluster so that we can use rsync um use the default and when they asking to kind of uh only uh synchronize the diffs between uh two directories or two data on datasets, uh but that's an overhead of maintaining the amounts, and you know a lot of things.

A

So it is that uh you know uh switch to uh using the surface client library, lips ffs and we do and we do support incremental uh um a synchronization of tata. uh But if right now in pacific, that's not there uh in the sense, uh whatever is available in pacific right now is like a bulk copy. It's like transfer the entire thing, um entire everything inside a snapshot. uh So if you are transferring another snapshot, so you delete the uh directory on the remote file system and then copy it again.

A

uh So a quick point about how a quick point about how you know uh the snapshots are synchronized for a given snapshot. Everything under the snapshot is copied over uh to the remote file system for that particular directory, and then a snapshot is taken with that particular name on the remote file system, so that ensures its snapshot. Names are in sync, so yeah. So right now in pacific, as I tell you know, uh it's a remote copy, but that is not that performant and it's slow.

A

uh So we have a pr work in progress that that that switches to use it based incremental uh uh incremental base synchronization based on inode uh m time and z time. uh So the code that I'll be showing you today that I'll be walking you through through today is uh the incremental base snapshot. One. uh The um the the bulk copy is not much interesting. The only difference in that is you know it's a it's.

A

It's a it's remove call followed by a bulk copy call, uh but we'll see how the incremental incremental uh synchronization is done so yeah. uh So, as I said, you know, this is it's a normal lips of fs client library uses lip surface drop to the local and the remote cluster, uh which means uh for for us for the image element to initialize the uh lip surface handle or to connect to the cluster.

A

It either needs the surf configuration file to be present on the host where it's running or it needs the mon adder and the mon address, and the key ring and the uh and the username to connect to the cluster. So for the primary cluster. We expect that you know the these have configuration file for the prime cluster is present on is available on the nodes, uh where the mirrored events are run uh for the secondary cluster.

A

We have two options, uh so the secondary cluster is is what we call as a pl to the primary cluster primary cluster file system, so that so we support two ways of adding up here. One is, via a pr add interface, and I think I covered it in the uh in the in the in the first uh talk. um So the pr ad requires the secondary clusters of configuration file to present on the primary cluster.

A

uh You know uh and the um key ring two to present of the primary cluster uh now there's another way to add up here, which is what is recommended and that's called boost. Bootstrapping appear is like bootstrapping appear on the uh remote cluster and then importing it on the uh primary cluster, so the bootstrap create creates a token uh which is nothing but um a kind of a digest.

A

You know uh uh hex digest of uh um the mini the the the the variables that are required to connect to a cluster which is like the monitor, address um the username uh and the um and the and it's keyring. So that creates a token and then we use that token to import it to import and to imp and import the token in the prime cluster, which is uh like adding appear again.

A

But at this, when you do a bootstrap based uh peer edition, you know you don't require the um the remote clusters uh contract yourself configuration file and the key rings uh so we'll see that how that's done okay uh and yeah so for pacific, we only support uh running a single mirror demon. uh You could possibly just run multiple mirror demons, you know, uh but that is currently untested.

A

uh So we recommend just running one.

A

And yeah so, let's see.

A

So it all starts up with you know,.

A

You know good old, uh main.c source cc source. We we do basic initialization here, just like you know, creating a messenger, uh uh creating a one client and doing some demonizing things.

A

That's probably the only thing we'll do here, not nothing much interesting here uh so yeah, it all starts with. You know.

A

Cluster watcher, uh so if you recall from my previous uh walkthrough um all the peer peer changes and the pr updates are stored in fs map and even the uh you know, what file systems are mirror are have mirror ring, enabled uh all those details are stored in uh fs map.

A

uh So the first thing that you know when I'm redeeming spawns is, you know, subscribe to fsmap because it needs to know you know uh what uh what is available, what file systems have mirroring enabled uh and what viewers are added, so cluster watcher is used for that? um It's nothing. You know it's pretty simple. uh uh You just subscribe to fsmap and you get notified when there's an fsmap update, which is here so we do a handle fs map we come to handle fs map, uh it's pretty simplistic.

A

You just walk through all the uh file systems available, nfs map- and you know you see, which has mirroring enabled uh you know if it's or mirroring disabled, which so it had mirroring enabled again earlier and now it's and also and now the mirroring has been disabled and you kind of figure out.

A

You know which file systems need to uh be enable mirroring and which needs to be disabled, mirroring well, for what I mean by enabling it you know uh internally, when uh when the meridian sees you know, yfs map that mirroring needs to be enabled it spawns a bunch of threads for those, uh you know and adds beer, and you know and and kicks off all the synchronization for that. Okay, so the way on this class is used and most of the classes here are kind of tried to kept.

A

You know uh kept so that it can be tested standalone. uh You know, so the way you use cluster watcher is by a wrestler listener, so you, you know, provide a listener, object and say you know. uh You know when when when something changed, which is like when notify me or call this, uh you know virtual function when you have mirroring enabled uh call this when you have disabled and psi nps are removed.

A

Okay. So before going into more details, you know, let's see some data structures involved here, uh we'll go into type s uh so file system. um Is uh you know a service file system which is like just a name? A string name, uh mirror demon uses uh uh uh a tuple of uh the uh file system id and the file system? Name?

A

uh Why we do. This is intelli when we, when, when we'll go through all the details, you'll see you know most of the things are asynchronous.

A

um You know so when you uh so it could be that a file system is, you know, getting deleted which is like equivalent to disabled mirroring for this from meredith's point of view, and it's trying to shut down, and you know back off and doing unmount and the file system gets created again with the same name. uh You know uh so because for that reason you know we we we not only.

A

We do not only use the uh the file system name as the id, but it's like a tuple of the file system name and the um the id. So the id is always, you know unique for a file system. So if you delete a file system and create it with the same name, it will have a you know a different fsid, uh so we use that internally everywhere.

A

um This is basically is just a specification for the file system. It's nothing but an encapsulated encapsulation uh with an additional uh pool id uh yeah, so yeah back to cluster watcher, uh you know uh so the subs, so the user of cluster watcher will actually when it's like initializes cluster watch object will give a listener and it needs to kind of you know, subclass this listener class and implement all these virtual functions. uh It's like you know, so it needs to implement, say what what do I do when mirroring is enabled?

A

What do I do when mirroring is disabled and likewise, for you know, the implementation when ps are added and views are removed. So that's uh what cluster watcher does? Let's see, if anything interesting, is there um yeah? So if you delete a file system, you know we consider that, as you know, mirroring is disabled, so we will try to back off. uh If you know we are trying to synchronize it, and here we actually go and call involves those virtual functions.

A

Okay, so where is this.

A

Used uh so all this gets initialized when uh so, if you, you would have seen main, then we actually picked things off by creating a uh an object of this mirror class, and then we call in init.

A

So this gets everything running up and running um in it. You know, does basic initialization. We come to service daemon, uh where we actually send updates to the chef manager. uh You know uh there's like a blob of data which, uh which is kind of like a status for the demon, uh so you can say you know these, so it's so blob. What we sent to the to the managers has like what files systems are. We are meeting enabled and what are the different stats? We we just include basic basic stats.

A

Like you know, we have this.

A

Many directories scheduled for mirroring uh these and we, if you encounter any failures if we recovered from any failures and things like that, so once we are done with all those um you know, we initialize the iman client um and uh that's pretty much it on the uh let's see if it's close to watch yeah and if you would have seen here, we do an init and then we run so we block on run uh run is nothing but you know subscribe to cluster watcher uh sorry subscribe to the fs map id cluster watcher and then you know in it, and then we wait until we are stopped, which is like a user terminate domination.

A

So uh once we are done with cluster watcher, everything is kind of asynchronous. Now now we are getting now we are getting these. uh um You know, callbacks uh cluster watcher is invoking these. uh These virtual functions, which are callbacks, uh and that tells us whether mirroring is enabled or not.

A

uh So what happens is let me quickly.

A

Show you yeah so this uh here we actually, you know subclass the listener class from the cluster water and we implement our own implementation, which is like what to do something to do when something is a mirroring is enabled disabled. Your ad and peer remove so yeah. Let's quickly, do you know what happens when mirroring is enabled?

A

Okay, so once moving is enabled um what we do is again. Things are kept pretty much asynchronous here. uh So how is this driven? uh You know uh once you subscribe to cluster watcher and then block uh until somebody, you know, does a termination uh we have these uh callbacks coming in, uh so these callbacks are actually queued.

A

um So once just like here, you know, if you see mirroring is enabled so we have something called a map uh between.

A

Yeah, so we have this map between the actions for our file system and the actions related to the file system. uh Mirror action is nothing but different types of action. uh You know it could be a mirror, enabled uh it's less like a list. um You know you can say for this particular file system. These are the queues of actions that need to be performed and all this is driven by a timer object.

A

Okay, so let's quickly see that too, uh so before that there are a bunch of config options that actually drive some of these things. So let's see that too yeah so yeah. So this has been converted to a yaml which is nice. So we have this surface, mirror action update interval. uh So this is like an interval for driving the async transmitter actions.

A

So when we are receiving these callbacks from cluster watcher, we queue them into a list and there's a primer thread that actually wakes up every you know default is two seconds you could adjust it if you want it to be faster and then executes these actions.

A

uh Okay, so all that is done um air.

A

So what we do is when we actually have mirroring enable for a file system. We we. We have these uh context completions that we queue, so the mirror action has something called as an action context. uh It's like a list of context callbacks, uh so we queue these context. Callbacks uh in that particular list and the the the update the timer thread just goes and just completes these context. uh So all this is again kept kept asynchronous.

A

uh So, let's see how animal medicine is done, which is probably the most simplest one when we get uh and we have measuring enable for a file system. We, you know uh this. This context gets completed by the uh by the updater thread and that calls you know just before this, but it completely com complete the context. So you end up in calling finish uh once that is done.

A

um We call enable mirroring. So we again come back here.

A

Okay, so here again you know what what we do is we we just initialize something called as an fs mirror class. So we have this fs mirror class that handles uh mirroring for a particular file system, and the file system is nothing but again, a tuple between the file system name and the file system id.

A

So we kind of initialize that and call you know initialize that particular class object, and once that is done, we'll see how fs mirror is uh is written. You know, uh um but yeah once that is done. You know we call this init and then we once once once the fs mirror initialization is done. uh It callbacks calls back here. uh So here again things are asynchronous, uh so once we do an init, we call this. We provided a context completion uh again, um and that on finish, is so.

A

If you see on this on, finish uh is nothing, but if you, if you see the class that we are seeing now.

A

This one when we actually enable mirroring so the context here is this. So when the context is completed, we call this gets called and we internally called uh handle enable mirroring so enable mirroring, which you know. We just move that uh particular. uh If the mirroring failed.

A

We do some things and you know, uh and and here we actually go into service daemon, and you know we see if, uh if things are fail, we update the service email so that that can be notified to the uh self manager and if we appear spending, we do an ad pair uh so yeah. So if you just follow this, you know uh for every particular action, uh there's a associated context class for it uh which drives the asynchronous behavior.

A

uh Likewise for discipline mirroring, you know restart. uh If you want to restart mirroring, uh which we think do at certain times when when for some reason mirroring has failed, we have this. um We restart. We try to restart it every uh not that frequently but yeah. You know we have a config for that. So when we try to restart it, we try to disable mirroring and then enable it again, so that again is driven internally by the uh by the context classes that we saw uh so yeah the takeover here is you know this?

A

All the baller plates here are basically just to drive, uh drive up a particular action for a of a file system which has mirroring enabled or you're trying to disable or you're, trying to add peers. All that is is put up here in this class. uh We do some other things such as. If you see these config options, we have, um you know, restart, mirror on block list and there's recently added. I think restart me around failure, uh so yeah.

A

So if you recall from my earlier talk, uh you know it could be that the mirror demons have been blacklisted um block listed sorry uh by the mirroring module that can happen. If there's a network of hiccup between the mirror demon and a network of which causes, you know the manager module to to. To think that the you know, the mirror demon is not responding to its um to its um to its notify messages.

A

um You can. It goes and block lists uh those addresses and once block listed, you know there's this again. uh You know once block listed, we we kind of restart mirroring again for those file systems that have been block listed uh like 30 seconds later so yeah.

A

That's again done by the updater timer thread here, which is like this part. It's nothing much uh interesting. You know just figure out from what what what's the next um action that needs to be performed and go and complete the context and takes care of certain things like you know, block listed, mirror instances and fail, mirror instances and tries to restart them. If you know if it feels uh so okay, uh now we come to uh the think the entity that actually does uh mirroring.

A

So we have this fs mirror class that takes care of uh mirroring for a particular file system. Kind of is multiplex for handling uh multiple peers. So right now we support only a thing adding a single peer, but you know we could probably support adding multiple peers too, uh so this is multiplex for uh for for supporting multiple peers uh by the way so yeah. So now.

A

The interesting thing here is um so, uh if you recall from my earlier talk, uh so there are two index objects that uh that's associated, that for with a mere demon, or rather not only the meridian, but the mirror demon and the manager module mirroring module. uh So we have this global um index object which lives which lives in the metadata pool uh for a particular file system.

A

uh It's called the cfs underscore mirror object and that's a global object, and then we have a mirrored daemon private uh object, which is like surface underscore, mirror dot suffix by the instance id, which is nothing but the rados instance id.

A

uh So we use all that here and we'll see how that's done um so yeah. So when we, you know, kick things off uh here we try to connect to the local cluster. um Well, the thing is uh for for the primary cluster. We need the ceph configuration file and the hearing for the user. That's that's! That's used to uh that's used to mount file systems in the um in the primary cluster.

A

uh Once that's done, we mount it and you know you know now we do this uh thing, for you know, ins uniting instance watcher, and you know so we have concepts of so. We have two things here. We have an instance watcher and a mirroring watcher.

A

uh The instance watcher uh class um is used to handle uh notify messages uh for the um me for the mid for the private uh index object, which is like the cipher sentence: computer dot instance id, uh and we have this mirroring watcher class. That's used to handle, notify mss for the global index object. The justice servers underscore mirror object.

A

Okay, so how's this that then, uh if you see instance watcher, um you know uh the crux of uh the crux here is: you know we we use the radar's aio notify an aio watch, or rather does the io watcher a watch api to establish a watch on a particular instance. um So here again things are kept asynchronous and this kind of a state machine. uh So what we do is first, we create an instance.

A

um So when we create an instance, um you do this area operation uh which is like you know, uh like you know, an object, write operation and op dot create. So this is like you can stack multiple uh operations on a particular object and then fire it off with a single aio operate. Call okay.

A

So here we do an ai operator which just creates so the moid here is nothing, but if you see the surface mirror dot the instance id and that's kind of uh initialized here in the constructor here instance of id instance id is nothing but the server object, which is like server sunscreen, mirror prefixed uh suffix by the instance id the router instance id. So we go and create that, and once that is done, uh we call register watcher uh register watch establishes a watch on that particular object uh for that we use the watcher class.

A

Register watch does an ai. A watch on that particular instance object. Likewise uh for the mirror watcher, uh it does the same thing, the only difference being you know it establishes a watch. It doesn't need to create it because the manager module creates um this particular. uh When you enable mirroring on a file system the manager module um it creates.

A

This ciphers underscore mirror object on the meta in the metadata pool, so we need to create that all we need to do is to establish a watch on that, so we call rest watch uh and once that is done, you know uh so whoever sends a notify for via that particular object. We get notified here where we call handle notify so for the instance watcher.

A

You know the notification message. This is not binary, uh you know because we send these from python, so it's like ascii based, so we send a json uh in the as as the notification payload and we you know, the json has basically two things all right: the directory part to operate on uh and the mode uh so uh recall from my earlier talk again. uh The manager, module kind of you know sends these acquire and release messages uh to the emerge demons.

A

So the acquire is like assigning a particular directory to a meridian and releases like you know, it wants the it won't. It probably wants to reassign that directory to another mirror demon and it wants the mirror demon which is currently synchronizing this directory to back off and not worry about it, not worrying about the directory anymore.

A

uh So we we have these. We get these json with the appropriate path and the mode. Once that's done again. We are all kind of listener based here, so we have this instance watcher where you know once uh once we get there uh on notification. We call these uh virtual functions, uh which are kind of um you know. Somebody has overwritten the the caller has always has subclassed this and and and overwritten these particular virtual functions for their implementation.

A

So we do that here. You know. We only support basic you in acquiring release modes is like riding coining a directory releases backing off, not to worry about that directory anymore, and this is all tied up uh in uh the fs mario thing. uh But let's quickly see customer.

A

Yeah, so we have this: uh we we uh we sub, we we kind of um derive from these from the instance watch watcher listener class. I call it snap listener.

A

um Okay, it should be called directory listener because you're not really listening on snapshots uh but yeah, it's kind of operating on snaps for the directory, so it's kind of yeah yeah. um It doesn't hurt to kind of rename this as teresa to be uh really consistent with what we are doing typing based um so yeah. We have this listener class again. You know we have the implementation, uh we we we, we overwrite the uh uh virtual functions and we and we call this- uh handle acquire direct when we want to acquire a directory.

A

Otherwise you know likewise what to do when a director wants to be released. So this is all again tied up in fs, mirror class um again so yeah. So this is when we need an fs mirror. So when we actually initialize an fs mirror uh object. When you create an object, I'm calling it you know. So it does all this it creates. The uh per private um index object, uh establishes a watch on. It um then establishes a watch on the global object uh and uh you know- and that's pretty much it.

A

I guess so this this can be followed by a state machine, uh a simple state machine, so yeah. First, we call uh init instance watcher that does the init of the instance what we saw once it creates the uh object and establish the watch. um This is the callback we call mirror watcher.

A

We initialize the mirror, whatever it is just like, establishes a watch on the mirroring object uh once that is done yeah once that is done, uh we invoke the callback which is like goes back into the mirroring uh so life, and once all that is done, we are kind of ready to handle any notifications from the mirroring module.

A

And what happens is we have these ad peer and remove peer interfaces so remember that the mirror module? Oh sorry, the mirror source kind of has this list of actions to be completed.

A

uh You know so one of these actions could be adding up here uh appear is, uh you know, uh is so we saw this in fs map uh last time. I'm not edge.

A

Yeah appear is nothing, but you know you have this. Uuid and some information about the remote endpoint, which is like the client name, cluster name and the fs name.

A

So when we get that, we kind of you know create free players for that. So you know it's like it's like. There are a bunch of threads assigned for appear uh that is controlled by the uh oh. Let's see, yeah service mirror mac snapshot. Sync, I'm sorry! Sorry, not that one.

A

Let's see where it is um like this can can't simplify it entry player.

A

um Oh sorry, that's in p, replay, okay, uh not retreats there, okay! So uh once we have that you know once we add up here, we call this inner triple a which which and which nothing is. But you know initializes the uh creates an object of the peer repair class, so peer repair, cla, replayer class, is where all the uh you know, logic for synchronizing, a particular snapshot uh to the remote target is implemented.

A

So uh here's uh you know this is actually multi-threaded, so we assign a bunch of threads for a particular appear so right now we support only a single pier. So, if you you know add up here, add multiple p, as you kind of insert here. uh You know uh you probably would want to remove this. uh You know sooner or later, uh but yeah so again. Remove pier does nothing, but you know you know whatever was initialized the peer repair class uh just go and shut down it. So nothing much in that uh again.

A

So when we get these handle acquired directory, we get these callbacks from you know, instance watcher class, which tells us that these this directory has been now. You know acquired this directory or released its directory directory kind of everything is kind of you know. You will see the service demon thing again and again in between in these places. This is like updating a a metadata blob which is managed by service demon, so that you know notes how many directories have been added for have been scheduled to this made.

A

Even uh you know, likewise, you will see in release it. Just you know, does the opposite, which is like you know it sets? Now we have these many directories to handle and the service demon kind of periodically. You know pushes all these metadata blob to surfmanager uh once we have once you know to acquire acquirer directory. You know, call this add directory interface in the preplay and we'll see peer replayer in a while, uh so yeah peer replay, um okay.

A

So the interesting thing here is okay, so I mentioned that you know um uh for connecting for for the mirror demon to talk to the remote file system in the remote cluster. It either needs uh uh self configuration file of the remote cluster uh and the key ring and things like that or you know we have this bootstrap thing um you know which so we create a bootstrap token on the remote cluster and then import it in the primary cluster.

A

What happens with bootstrap import is, I think that kind of we didn't cover this in the last talk, I think we didn't reach that so I buying jr mirroring snapshot boot up import.

A

Okay, so, let's see create is nothing but you know you you, uh we create a token uh which has uh so the token would have you know, v basics for encode x, encoded, but it would have the um the cluster uh fsid the file system name, and this needs to be done on the remote uh cluster. It would have the remote clusters fsid the fs name and the user, so you create a user here. We use the fs authorize to create a user, and then we fetch the keying e, and we have this again.

A

Sorry, okay, so it's just a dictionary. Based of uh you know, all the minimum things that are required to connect to this particular uh cluster and file system and the mon host once that's done, we kind of uh you know, uh create a basic civil encoded string for that, and that becomes the token and the interesting part is when we import it in the primary cluster. We just decode the token uh once that's done, we have the required details, uh we call into prad.

A

So what what's done is all these uh you know the the um um the the keyring and such are kind of uh stored in the mods, uh so it uses the mon config store. For that, let's see, add: okay quickly, um we do some basic checks. um We covered this. We kind of try to set up. You know uh uh an exciter on the peers, uh root uh file systems root so that you know if it's, if two gun two two clusters try to do, uh try to do a pr ad on the same peer.

A

You know only one succeeds because it uses that excited, create uh you know, flag for the metal trading and etc once that is done yeah. So this is the point where you actually go and do a current and do you know the so, uh for we do a config set.

A

Yeah, so we save these details into the man config store uh we, the key here is uh we have this um peer, config key, which is like slash uh via configure peer file systems and the pro uid uh so recall that you know every peer has a uuid associated with it, so that becomes a distinct key for that particular appear and we save all these details. So if you see where config set is done, we set these config set file system and do a set of the remote conf remote config.

A

Basically, we just have the mon host the user name, the key and something called as a site name site name is just use. You know a user string that can be anything. That's used to identify this particular remote site. It's not used anywhere um internally in the mid events or in the manager module it's just for the user to make sense of okay. So uh what we do here is we try to connect in back to the mirroring module. uh So we try. When we initialize the spree player, we want to try.

A

We try to connect to the remote uh cluster and mount the file system, uh so we do. We try to fetch these uh required. Details uh creates from the monster man config store. uh You know we do a one command, get those keys once. If then, if you get something decode it and then uh for this connect interface, we provide the mon host and this fx key. uh We, the client name, uh you know the the host and this fxe um and.

A

And you know it's nothing, but but you know once we have uh those monos and keys, we set them in. We overwrite those in the uh in the ccd con subcontext cons. So here we set the mon host and then the key. So with this we do not require the remote clusters. Config uh left config file and the key rings to be present on the uh you know, as you know, in edcsaf we don't require that we do with the bootstrap.

A

We can override that and uh save all these details in the man, config store and the self made even reads it back and uses it.

A

Okay. So, as I said you know, every peer has a specific number of uh threads assigned to it. So you have these multiple threads, trying to synchronize uh snapshots, folder directory um a thread handles a directory. So it's like. We have a list of directories uh for which uh the snapshots need to be synchronized.

A

Each thread is associated with the directory instance. um It tries to pick pick a directory uh for which snapshots have been synchronized, so you can have 10 directories and just three threads, but each of them will have just just handle one. So it's not we.

A

The thing that has been taken care is, uh you know uh two or more threads cannot handle a particular directory because in the end we use you know the incremental. Basing you know that does snapshot one at a time uh or even the you know, old-fashioned, uh remove and bulk copy, just one snapshot at a time uh if multiple threads were to synchronize snapshots, you know that would probably be bad.

A

That would be disastrous, because multiple threads will be copying, uh snapshot for the same uh directories to the remote file system, which is not great, so we do snapshots one at a time for a directory.

A

uh Let's go back to.

A

uh We are seeing this max congruent directory, syncs, okay, first one yeah, so yeah we we default s3 would probably increase it based on you know how how big your machines are and how fast you need uh synchronization, uh but the the main point is we have a bunch of worker threads uh for handling a snapshot synchronization.

A

So we have seen these interfaces or rather seen you know the fs mirror class called these interfaces. uh uh The ad directory, just you know, adds it in the queue removes. Just you know uh is the reverse is like you know, just remove it from the cube, but if synchronization is going on, you know you ask it to back off, uh you know, and then we remove it uh most of things. Here are pretty straightforward.

A

uh You know not. You really need to go into each implementation. Details. uh There's one thing that we do here is: you know, once a thread picks up a directory for snapshot. Synchronization, uh we lock, we do a.

A

um We do an f, uh f, lock on that directory on the remote uh um file system. uh The the the the reason we do. That is, uh you know recall that you know we probably want to support multiple surface mirror daemons synchronizing uh congruently at some point of time uh so say a particular directory path is assigned to a mirror daemon and that some of these threads are one of these threads are kind of actively synchronizing direct snapshots for it. uh You know, another demon comes up and the mirror demon.

A

You know figures out that this particular directory is, you know we can probably shuffle it to the other mirror daemon, so that uh for balancing- uh and it happens that this this uh uh mirrored image current- this mirror daemon- is for that value. File system is currently synchronizing that particular directory snapshots in the directory, um and um you know when these acquire and release calls. You know they are not blocking calls right.

A

So release call may come in and you know uh internally when we actually showed you the removal of a directory uh or at the release of a directory. uh We act back saying that you know it's done, but internally, if a synchronization is going on, you know it needs to back off later we do not block uh before we can. uh You know release that particular we finish or interrupt uh synchronizing that directory uh so from from the mirroring model from mirroring module point of view.

A

uh The mirror daemon has kind of uh you know noted that this directory needs to be released, and now it can go ahead and you know assign it to another mere demon and that mirror demon uh now notes that you know it's it's uh you know it needs starts to it. It needs to handle the snaps for that directory.

A

um So the f-lock here ensures that you know we don't run into a case where we are backing off uh from amir demon is backing off synchronizing for a directory, uh while the other mirror demon starts synchronizing that a directory uh so before we, uh you know, uh scan snapshots and uh synchronize them. You know we take and we take a. uh uh We take a lock exclusive, lock or a non-blocking lock uh on the remote uh on the director on the remote file system.

A

So if the other mirror demon were to, you know, try to pick that directory and try to synchronize it, but this mirror daemon is just backing off. It wouldn't have released the lock or just about to release log. uh So we don't run into a case where you know one mirror. Demon is kind of uh backing off uh completing a particular io operation and the other merit even kinds of is: does the ia operation again? We don't know what will happen uh so we got all those by an exclusive uh lock and it's non-blocking.

A

So if we they, if it, if the flock all uh returns and that we we might block um you know we retry it later, which kind of tells us that you know it's probably locked by another mirror demon and sooner or later it will probably release it, and then we acquire the lock and do the normal operation again.

A

uh Once that is done, uh you know, that's that's the main part yeah. Then the other interesting thing is we uh yeah before for for a particular directory. um You know so the email demon can identify snapshots that have been deleted and renamed. um You know. So you can have some of these snaps. You can rename a snap, uh you know and delete some snap and you know, mirror demon will, can identify that you know if you rename a snap, it's not like limited even would delete it first and then copy the again copy.

A

It again copy it again. This is all based on snap ids, so we have an interface to fetch the snap id for a particular snapshot.

A

And with that, we built a map key being the snap id and the value being the snap name, and we do this for the local and the remote file system for our directory, and we can figure out. You know um if, if, if a snapshot yeah so before we build the snap, you know. uh So how do we know that you know a snap has been uh deleted because the snap id is on or rename because snap ids on the remote file system for a directory would be different because that's another cluster.

A

So what we do is uh when the mirror demon synchronizes a snap.

A

A

We use the mk snap call to create a file system, so if, when it has, you know kind of uh synchronize all the data, we do a sfn case snap, and these are the things we added for numerity when we added a snap metadata, uh uh which is uh it's just a free-flowing key value pair, that you can store uh that that is actually attached to a snapshot.

A

uh You know the um it's just like a you know: free flowing key value pair tap in the data, and then we have the snap in for which, which is a call uh you know which, which can give which it gives you, the snap id uh and the snap metadata if any, that was associated associated for the directory for the for the directory snapshot when creating a snapshot.

A

So what we do is uh so we'll see here, um let's, let's.

A

Okay, so what we do is, uh while creating a snapshot once you have transferred all the data, um you know you, you stole the uh for for the snapshot that is being transferred, that is being synchronized. It has a snap id on the primary file system. uh We stored this particular id um in the metadata on the remote snap on the remote snap editor if it had id2 on the primary file system.

A

When we take a snap, the snap on the on the remote file system can be anything can be, 10 can be 12 any number uh but to to to identify these deletes and renames uh and to and and and to ensure that we start from the correct snap. uh We store the primary file system, snap id for that snapshot. There's like two here on the metatara on the remote snap. So when we do this build snap, so when we do build snap.

A

So when we build this map map, uh you know for the primary file system when we are building this snap app, which is less like a map between uh so we'll see here.

A

A

um When we build these map map you'll see it's just a map between any dj and a string, integer being the snap id for the primary file system. It's the actual id the snap id which is, which is the idea of the snapshot for the remote file system.

A

It's the id from the metadata that we stored um so it so. The id here is not the snap id of the of of the or the snap on the remote file system for that directory, but the actual uh id of the metadata which is id stored in the metadata, which is nothing but the uh the id of the snap id of the primary uh sniper id of the snapshot in the primary file system. uh So once we do that, we now have this map.

A

uh We can compare this map, you know if an id is if a snap id is missing uh from the local snap map, but I live in the remote snap app. That means it's deleted uh if the id is same, but the name is different. That means it has been renamed uh so it so we can just do a you know. A rename call on the um on the remote file system and read them that particular snapshot.

A

uh So once we do this snap map uh now we need to figure out uh so so this is uh so this code. uh What we have here is is the incremental base. Synchronization that is now under review as appear the master code is, would be a bit different because you know the incremental part is not there and we do just the rmd, remove the contents and then copy the whole thing again, but here we have this incremental snap. uh So what we do is um you know, based on the snap map.

A

Looking at the snap map we figure out uh was the next snapshot to start synchronizing from um that is that you can infer it from the uh local and the remote snaps. uh Once that's done, uh we check. If uh we could. Probably uh you know, choose incremental snap uh now. How is that done? So what is done is uh so we have a new exciter for this. um Let me quickly do that yeah, so we so in that pr uh introduce this f dot, mirror dot dirty snap id.

A

So before we start synchronizing data for a particular um for a particular snapshot. uh We uh we we, we set this exciter on the directory um on the remote file system. That is the directory of the. That is the snap id of the snapshot that is currently being transferred.

A

uh So uh initially when, when we are transferring the first snap, this etc is not available, which means you know if the exciter is not available, we can we we need to do well copy, because there is nothing to compare to which is like equivalent to starting with the first snapshot, so that needs to be able copy, uh so the bulk copy, before starting the ball copy, we'll do the uh we'll set this exact on the uh remote directory, uh we'll cop we'll do the transfer and then we'll do a snap uh when we come back and choose another snapshot uh to synchronize.

A

uh We do this comparison based on the uh what's available as an x as the etc value in the remote directory, and what are the snapshots? What is the snap id we're?

A

Transferring from uh so, if there's a match between the snap ids um on the remote directory, which is the the value of the exciter in the remove directory, uh if it matches one of two, if it matches the um the the uh so with incremental snapshots, we have uh what we do is um we'll have uh we pick two snapshots so one that is already being already transferred to the uh remote and the other that is currently being transferred, so they will have snap ids.

A

uh So if um so, if the excited value on the remote directory matches one of these snap ids, we can do incremental transfers based on local comparison, which means we. We need not compare the current snapshot uh data uh with um uh with the remote uh file system data for that for a directory. We can infer it locally by comparing uh snapshot by comparing two snapshots uh which are in the local uh file system, and that's because the uh snap id tells us that you know the data those.

A

So the idea of the snap id uh the the the value of self mirror dirty snap id is: it identifies the um the the it tells which the data under this particular directory is associated with what snap id, so it could, if it's associated with the snap id of the older snap, uh it's fine because then again we can do local comparison uh between uh the earlier snap and the next snap.

A

If it's the uh so it could happen that you know we are transferring data for a particular snapshot which means setting the snap id um for the dirty snap id, etc. And then you know the cipher is mirror restarts once it restarts. We get this snap id which now matches the current snap id. You can again choose.

A

We can again do uh incremental transfers based on local comparison, because the snap id matches uh we know that the data on the remote is now somewhat incomplete, but it's, but it belongs to the the data for the current snap id um the catch here is we need to ensure that the snap vr, the snapshot we are comparing from is also you know um the the data was transferred uh to the remote file system based on. uh We need to ensure that before we before uh uh you know the surface mirror demand was restarted.

A

For some reason, uh the snap id uh the the snapshot, the data on the remote uh file system is is is transferred by comparing two snaps and after uh restarting we are still comparing the two snaps that needs to be ensured, because if we choose a different snapshot now you really can't trust the data on the remote files on the remote file system directory uh yeah. So we do all these text. All those checks are based on the dirty snap, id etc and the local and the local and remote snap maps.

A

So with this, we can kind of figure out that you know uh if it's safe to uh go ahead with uh incremental transfers based on uh local comparison.

A

If, for some reason we cannot say we are transferring maps and we are transferring a snap by comparing to snapshot x before, which is a snapshot that was transferred, uh snapshot x and now snapshot y, uh then the server's mirrored email is restarted or the mirroring is disabled and then, when it comes back again, you know so it was mirroring, is enabled again the snapdex doesn't exist.

A

Nah snap x doesn't exist, so you know if you're, comparing to the snapshot before that, uh so you know, obviously we can't use uh incremental snapshot based on local comparison. In that case, we use incremental snapshot based on remote comparison, just like comparing the snapshot uh in the primary file system, with the actual data on the remote file system. Under that directory uses the same logic. The the logic is same.

A

It's like comparing um data between two snaps or comparing data between a snap and um the data on the remote file system directory.

A

You know the the logic is same is just that we use the uh the um the correct uh mount variable, which is like either points to the local uh local file system, and the path is the local is the last snapshot or it points to the remote file system and the path is the remote file system directory and all these is again. You know we based on the recently introduced, or rather almost uh ready to merge at calls based apis.

A

uh So we kind of open an uh file descriptor on the um you know, snapshot directory and do um add base calls uh relative to that with lefty. So if you're so- and this is not yet merged- uh so um you know it could happen that when we are transferring a snapshot, uh you know uh somebody deletes a snapshot created with the same name but entirely different contents. You know, uh if you do path-based pure path-based operation, we'll kind of you know there might be a case frame.

A

You might, you know just uh uh transfer the incorrect data uh because we are just doing path based operation, uh but with all these are all fd based uh ad based calls. You know you have the the the inode pin in memory, because we have that every open- and we do everything related to that.

A

So this yeah prt player is so that's the most. You know uh thing to be discussed about pre player. You know the the crawling of the file system data is you know uh it's nothing interesting. We just crawl, uh do a walk of the file system and then compare and sync yeah.

A

uh I think I covered most of the parts. uh Let's see, um service demon is pretty simplistic.

A

You know just uses um the uh manager, or rather the aps, provided by radars to to to periodically update data, uh the metadata block to self manager and we kind of show- and we have an interface in the manager module to kind of you know pretty print the uh the json, the blob which is sent by uh these mirror demons.

A

I guess that's it um yep.

A

Thanks guys. uh Thanks for attending the uh code, walkthrough.