Ceph Ceph Virtual 2022, 15 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DisTRaC: Accelerating High-Performance Compute Processing for Temporary Data Storage

Description

Presented by: Gabryel Mason-Williams

DisTRaC: Accelerating High-Performance Compute Processing for Temporary Data Storage

There is a growing desire within scientific and research communities to start using object stores to store and process their data in high performance (HPC) clusters. However, object stores are not necessarily designed for performance and are better suited for long term storage. Therefore, users often use a High-Performance File system when processing data. However, network filesystems have issues where one user could potentially thrash the network and affect the performance of everyone else's data processing jobs in the cluster. This talk presents a solution to this problem DisTRaC - (Dis)tributed (T)raisent (Ra)m (C)eph. DisTRaC offers a solution to this problem by providing a method for users to deploy Ceph onto their HPC clusters using RAM. Their intermediate data processing can now be done in RAM, taking the pressure off the networked filesystem by using the node interconnect to transfer data. In addition, all the data is localized, creating a hyper-converged HPC cluster for the duration of the job. DisTRaC reduces the I/O overhead of the networked filesystem and offers a potential data processing performance increase.

Learn more about DisTRaC: https://github.com/rosalindfranklininstitute/DisTRaC
Event: https://ceph.io/en/community/events/2022/ceph-virtual/

A

So um I'm Gabriel, Mason WIlliams and today I'm going to be presenting distract, which is distributed transient rancer for accelerating high performance, compute processing, the uh temporary data storage. So before we get into that I'm going to give a little bit of background about the Russian French Institute and who I am so the rose and Frank's Institute is a United Kingdom, Research Institute dedicated to developing new technologies to tackle important health research challenges, we're based in Harwell uh campus and didcot, and we're funded by the UK epsr DP ESP RC.

A

However, we can situated with five themes the artificial, intelligent informatics, biomass, spectrometry correlated Imaging in this generation, chemistry and structural biology, all of which generate an awful lot, an awful lot of data and then myself, uh I'm, currently a master's student in artificial intelligence at Queen, Mary, University of London and working as a research software associate at the original Frank Institute I'm. Also uh looking for a PhD so back home to the project in this project has spanned many years and has had many.

A

People's involvement originally started a diamond light source where I was inspired by Mark Basham and Dave Bond, who came up with the idea behind this Dennis had involvement with the fcfc staff user group and then recently the University of Bristol of Psych collaborating with us on it. So today we're going to go over the background to the problem, learn what the problem is. uh What is this track?

A

Why diss track and not another deployment tool, how distract Works, how distract solves the problem presented, distracts workflow and then I'm going to present two case studies where we've used distract to improve the performance of a HPC applications and then leading with their conclusion and then time for questions so background's the problem. So this is how a traditional HPC cluster is set up. You have a job scheduler which are schedules uh different uh applications onto the um The View cluster.

A

So if we had a job that required two nodes, it would wait for there to be the relevant room and then allocate said nodes. Then you have the compute, which is often comprised of lots of RAM. Then you have the storage, so essential uh storage facility. Here that can be any um a file generally a file system. Then you have the networking which is comprised of nodes interconnect, so the nodes can talk to each other and then the nodes connection to the high performance storage cluster.

A

So what's the problem with this? Well, there are a few so for I o bound applications, they're limited by the network connection to the high performance, storage cluster. It's also a shared resource. Therefore programs can they can process that Library are able to thrash the share to file system which can interrupt or disturb other users jobs accessing this file system.

A

Also, any data that is stored is stored here, so intermediate data- that's made during processing and has to be written here, gets data for the long term. So the issues here is that we could have node one say this is running at line rate and thrashing the the cluster. This will then affect node 2, all the way to node n, which is not optimal, and it doesn't provide good performance.

A

So what's the solution today, I'm proposing diss track as the solution. So what is distract diss track announced the creation of a distributed transient, Ram stuff cluster. It's a program for deploying a transient run soft cluster onto a HPC infrastructure, utilizing the ram in a scalable and efficient manager, essentially we're creating a job assistant and isolated in-memory file or object store for the job application, so distracts just a deployment tool.

A

So why not just any other deployment tool, so the other deployment tools that are developed in um saf are often quite slow as there's lots of safety checks required, they're sequential they would be over engineered for this use case. The requires a sage and they're designed to build long-lasting, maintainable clusters, which are not what we need right now for this use case. It's going to last for as long as the job requires it's inherently volatile as we're building on top of ram.

A

So if a node goes down, the process stops anyway, so we needed something that could you know be quick most of the time as most of the time using HPC cluster should be used, processing, not saying it out for storage cluster.

A

So how does distract work what's different about its deployment method? Well, it takes advantage of the homogeneous structure of a HPC cluster, and what do I mean by that? We can use MPI. So the distract level of deployment is that it will deploy the minor node on the head node. It will then place the keys on a network file system. So, instead of needing to SSH around and copy the keys, we can take advantages that there's an uh a NFS file system present such that once it's placed on Node 1, all nodes can see it.

A

Then we create the management node, and then this is the point where District gets the mesh speed up is that we deploy the osds in parallel. So the OSD is here on this. Node are also deployed at the same time, across all the nodes, as every node can see all the keys required.

A

This means if we want to build six osds on this node with each containing 30 gigabytes of RAM. We can do that and that will happen across the whole way through and then we can then create the the pool, the NDS or the release Gateway and then Mount the file system, if it's requested. But what does this mean in practice? In practice? It means that the deployment time is roughly 22 seconds or so, uh as you span across so here we scale it from one note to six deploying one OSD per node.

A

Of course, the more host easier, um the more osds that you've set on each node or span the deployment time will slowly increase because of the way that Seth allocates pgs, but um generally very quick.

A

So how does the removal work? We take the same process we apply NPI across for removing stuff, as we know that all the nodes have stuff. So um this is fine. This is a secure, secure operation. We can then remove all of the osds. As we know, all of them have the same amount of rsds and the same structure, and then at the end we remove the files from the NFS, so they can no longer see and they no longer exist and then.

B

A

End and how does this look in practice? Well, we also get a removal time that is about a minute and a half so again, very quick and very efficient. This means that we're not going to be eating into the processing time of setting up the cluster to access these things, so we can now face distract uh deployment and removal within the job submission script, to create a ceft cluster in a fast and scalable way that lasts as long as the job process. But how does this solve the problem?

A

So the problem now is moving the I o bottleneck to the node interconnect so we're now taking advantage of the interconnect between the nodes, integrate. The storage cluster we've also isolated resources, so these are only accessible from the compute node schedules within the job. So if this job was able to thrash the loan rate of the previous storage cluster and writing to it, it will now only impact this environment.

A

Here it will not impact other jobs on the high performance storage also, we can now make sure that all the intermediate data processing happens across these nodes instead on the group. Instead of the global storage, reducing the I o overhead of the HPC application, so if we look at two case studies now we're going to explore rely on anxiety.

A

First of all, what is rely on is requirement microscopy, structured determination, program used to the ocean fragment tube Iran uses a whole node cluster allocation, producing small intermediate files and connected to the impact of the usage jobs running Elsewhere on the cluster.

A

So when benchmarking this, we when testing this and comparing it against a diss track and resulted in a processing time or reduction of 5.51 and a total reduction time of 4.37.

A

So the total reduction time is lower than the processing time, as it was fairly fast to run, and the deployment of districts then eight into the performance. uh The performance time also. uh The reason why rely on performance here isn't incredible is that it's a mainly a um compute band process. So the main advantages that we game hit gain here is that we remove the need for a high performance file system in the cloud as we can instead just deploy it using distract.

A

Then we have case study uh solving. So cyber is a tomography, reconstruction and processing pipeline use, the diamond light source and the erosion fragrance machine. It runs using hold node allocation and can run at line rate producing intermediate files as it as it goes. It's capable of saturating access to the parallel file system, which can negatively impact other users jobs running after on the cluster.

A

So a normal side view process is demonstrated in this figure here. So what we do is we place in the initial data and it goes into saving. We then do process, one which is dark flat field creation um correction, which is then placed on the high performance storage.

A

We then go over to process two, which is Raven filter again based on their and then process, three, the Pagan filter, and then we have the last step, which is the most important step that we care about, uh which is the Reconstruction, which then goes to the process, data which can also be on the high performance file system. Now this data generated here is not relevant once we get to step four, so once we've done step one, it's no longer relevant once we're into step two and we start processing on it.

A

Then, once we've done step two uh step three step, one step two step three, and then this final step is the one that's important, and this is the intermediate data. So if we run it using side view does not distract, we can instead make sure that all the processing for the intermediate data instead happens using District set as represented here. So now the processing happens on the set within the nodes. It's tough within the node separ than the nodes, and the last step which we care about is done on the global high performance file system.

A

This results in an overall processing time reduction of 8.32 percent. However, it reduces the I o overhead by 81.04 percent, whilst utilizing the same amount of nodes within both setups this uh test and benchmarking was done at Diamond light source.

A

So, to conclude, this track is a self-deployment tool that creates a hyper-converged HPC cluster for the duration of the job. By utilizing the ram of the community as a result, diss track reduces the eye overhead of the network file system and it offers a potential data data processing performance increase, depending on the file system. That you're, using underneath the other unique um advantages to this method is that we can now select whether we want to use an object, store or a file system.

A

So this last step can instead be writing out to an object store instead of the global file system or a remote store. Instead, these are the references. Thank you very much for listening without any questions.

B

A

Somebody has a question in the chat. um Stefan says: do you use sufferfest for this, so we use Steph natively. For this other example. We use stuff natively, though using rados as the connection mechanism communication mechanism and then uh for the rely on we use ffs.

A

So um savu is mainly dealing with image data, so our object stores is a more efficient mechanism for dealing with the data.

A

um At the moment we don't scale the mdms MDS service on each node, but this is something that could be added.

A

uh So the pool size we uh it's so I've, it's implemented to it auto scales, so the user doesn't need to state or have any information about the pool and it's done on the amount of osds. But essentially the replication is set to zero. The uh the pool size isn't the most important factor in the performance because, uh because we're running on RAM, we don't care about whether um the system goes down, because uh if one node goes down during the processing, then that processing ends anyway, because it's utilizing NPI.

A

um So local nvme's could be used. uh We haven't implemented it such that it can deploy it onto local nvmes as well as we haven't, had access to any nodes to do that development, as we need to make sure that the uh the nodes uh the local nvme's aren't currently being used, whereas the ram it's a lot easier to do that check.

B

um I guess one part that will mean an answer is uh how much RAM uh like in total. uh Would you typically need on each node uh for, for you know to to do this efficiently, so.

A

It depends on the process that you're running so for um the side view for that application. It we used 80. um 80 GB of RAM on each node. uh So that's that was approximately half the ram that was on the cluster node. We don't often go over half the amount of RAM on the clock on each cluster node uh for stability reasons, as the the processing also will require some.