Ceph Ceph Virtual 2022, 14 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Accelerating PMEM Device operations in bluestore with hardware based memory offloading technique

Description

Presented by: Ziye Yang

Accelerating PMEM Device operations in bluestore with hardware based memory offloading technique

With more and more fast devices (especially persistent memory) equipped in the data centers, there is great pressure on the CPU to drive those devices (e.g., Intel Optane DC persistent memory) for persistency purposes under heavy workloads. Because there is no DMA-related capability provided by persistent memory compared with those HDDs and SSDs. And the same issue also exists in Ceph while using persistent memory. We would like to address such pain points by leveraging memory offloading devices (e.g., DSA). So generally in this talk, we will talk: 1) While persistent memory integration is not very successful in Ceph due to the high CPU overhead while performing I/O operations on persistency device; 2) We introduce the memory offloading devices (e.g., DSA) in order to offload the CPU pressure while doing I/Os; 3) We will describe the main change in pmem device (i.e., src/blk/pmemdevice.cc) and state how we can achieve the offloading including the challenges. 4) We would like to have some early performance results if Intel's SPR platform is available to the public.

A

So my name is yeah. It is my great honor to present this topic in uh self-outcome virtual conference this year.

A

So I think my presentation can be divided into the following four parts: the first, where is the background and the program? And secondly, we will introduce our proposed solution, that is, to leverage Hardware based offloading technique to mitigate their uh CPUs bottleneck when operating a persistent memory devices and in the third part we will introduce how to enable DSA dsa's phone name is data streaming accelerator in depth and finally, we will have the conclusion.

A

So, let's see the first part background and the problem. We know that with the trend I mean that there will be more and more deployment of non-rite memory devices, for example, uh interest often Optimum memory, so I mean there are some strategy changes, but it will not change the trend, and, secondly, is the CX here attach the persistent memory. I think it will be available soon in the near future.

A

So it means that, with those device deployed in data center, I mean that CPU resources on the host will become the bottleneck while operating on those persistent memory devices, especially, for example, with the application direct mode usage. It means that you can treat those devices which is which are attached in the ding as a persistent positions, the storage.

A

So it means that for this usage it is a different compared with operating uh pcie, SS, PSI, ssds or other such ssds. For example. I can use the diagram in the right part as the example you can see in the left part it. It is a high hyper converged infrastructure. It means that Computing resources, standards and their storage and resources are deployed together.

A

So you can see that when the system are attached with the pcie or susts, when we run self the host the CPU described I I'm fine to serve 20th virtual machines, because my brothers, like the pcie ssds, they have the dma ability to offload the the read write of operations from the memory to the devices.

A

So it means that CPU can have a bunch of uh plenty of resource CPU resources to execute the virtual machines. But if you uh change this situation, it means that you replace those ssds with the persistent memory devices.

A

The CPU May complain that no I can only serve 15 virtual machines and are very busy because there is no uh dma direct memory access capability from those persistent memory devices. So it means that in such situation, uh without the dma capability from the devices, CPU will be much busier to the other real write iOS to those devices.

A

So we we conclude that the root causes uh is that the data operations between memory or persistent memory is different from the data operations, uh G memory and the devices, because for the later usage case there is dma or RDMA capability on those devices. So so there will not be too much uh button on CPU resources, but when we use the persistent memory like devices, it will uh have more pressure on the CPU resources.

A

So we think that Hollow can we offload some work from CPU with some Hardware offering technique next, and we think this is a valid requirement, especially for those Cloud providers.

A

In hype, converged infrastructure, usage mode, it means HDM mode. It means that the cloud providers would like to utilize the important CPU resources in the host to serve the customer, visible workloads, for example. The virtual machines and the workload in the inside the virtual machines, instead of using a more CPU Cycles on persistent memory devices yeah.

A

So it means that if we can save the CPU resources on the persistent memory- and there will be much more CPU resource remainder to run more virtual machines and those Cloud hypervisors, they were happy to such usage.

A

So I think this is a very common problem and if we yeah probably many persistent memory devices in surface OSD, for example, Brewster or system, we will face the same issue written with CPU utilization.

A

So we we have, we will proposed our solution.

A

So we propose to leverage some memory off-roading devices, for example, there are already some memory off-roading devices provided by India, for example, India's io80, I o x, a region technology and the upgraded version India's DSA data streaming accelerator, for example.

A

A data streaming accelerator is a high performance data copy and transformation accelerator that will be available in India's next generation of processes, for example the cephal Rapids, and it can be used to Auto optimize streaming data movement or some other operations, and it can be used in many different scenarios in storage, networking and also persistent memory. For the example. The most popular operations can be the move, operation move. It means that you can use the move to move data from persistent memory to memory or memory to memory.

A

It is it can it's it's all, it can all be supported and there can be other operations like the CRC operations or field compare Fresh, and in this presentation we will use the move operation and.

A

Off-Roaders are memory uh memory and their the movement from memory and the persistent memory.

A

So I mean that there are still problems how to efficiently leverage those memory off-roading devices.

A

We think that we can have two different models to interact with those uh memory offloading devices, because there are already uh support from kernel or user library to to integrate those devices. So it means that if we want to replace the memory copy operations or movement operations from memory and persistent memory, we can just replace the CPU CPU statement with the operations dedicated by the library and then we can use it. But hey the and the first usage is the sync mode. The sync mode is that you will not change the behavior of the application.

A

For example, instead Brewster you can just replace the the operations uh provided by the pmdk library. It is very simple and we know that the integration overhead is low, but the advantage will not be clear in CPU utilization and the performance main may not be improved for the workloads, because when you use the DSA to offload the operations you, the programs still need to wait their results, so there will be no CPU CPU side, resource decreasing and the second second usage is uh I think a sync mode, a sync mode usage.

A

The integration overhead is very, very high because you need to rewrite the the programs, for example, is a very famous uh approach. Application work is the dvdk or sbdk and itself. We. We now already conducted the tistor working, which is uh derived from a system framework. It is also a sync mode, but I mean that in Brewster in the persistent memory device uh devices Library currently I mean that if we want to change the a sync mode, there will be still additional work and it is still in a progress.

A

So I mean that for this integration, overhead is high, but I mean that if we use this, the CPU resource, saving and the web is uh definitely a performance. Improved. Improvement when we operate as a persistent memory devices in sales OSD and the two demonstrate that our idea is practical, we provide our solution itself. Currently. I think that the sync mode to leverage the essay I mean the code is already itself's Master branch and there we use uh India's, DML, a data, mobile library to javdsa and, firstly, accelerator persistent memory operations in cells Brewster.

A

So this part we will introduce some details. I mean how to enable DSC itself, so I mean that currently we divide our work into some steps. I mean that contains the mainstream or work itself is system, but but I mean that the work is still in progress, so we so we will integrated the work in Brewster first and the then we will continue their work in in this store in the future and in phase one.

A

We already have some achievements to off-road to offload persistent memories in Brewster and we leverage India's data, move library or idxd config Library, idxd, config library is a tool to configure the DSA devices in the host. If we, if you have a India's latest spr platform, and even we enable the uh TSA configuration itself with the configuration, if we do not uh configure the DSA devices in the host, it can still, uh we can still run the self with the software pass provided by our approach.

A

So that means that it will not hurt the usage of self. If you configure DSA enabling, but there are, there are no dsh devices in the host another step. One is to use the sync synchronized apis in DML library to offload CPU operations, and the expectation is the functionality. Functionality can work and we already achieved this I.

A

Remember the the following: two patches are already merged, maybe in the middle of of this year, or maybe I I mean they are in uh in April or in April, maybe yeah, and the first patch is that we we did some work to support the DV, DX usage, amp system memory and in the next Pages we will introduce. Why we want to this, and the second patch is that we we just Leverage The.

A

Synchronized apis in the DML library to operate on the system memory, and we already did some uh evaluation uh in in our own environment and it works well, and the second step is that we propose to use a single, a single API in DML library to upload a CPU operations, and this work is doing in progress. And, firstly, we will rewrite the uh I realizes the implementation, details in blue stores, pmm device implementation and that changes are think of behaviors.

A

On persistent memory with I think, then we will uh replace the CPU operations in a sync mode with the off-roading devices, and the expectation is that we should have the performance Improvement and mitigate the cpu's pressure. But currently I mean that, according to a sales architecture on Brewster I think this. This work will not be will not be easy. It is different with the applications which use spq application framework, because I mean that in SB case a sync framework. We already enable the uh the essays with some operations.

A

For example, the CRC I check some Emmanuela TCP workloads, and we already have the very huge performance Improvement, which is already demonstrated in this year's India's. Innovation event, and we also have their the POC code.

A

It is not emerging itself in sbk's master Branch, but it is a variable uh on sbk's called a code review website and it also demonstrated their performance Improvement yeah.

A

We know that there are two uh possible usage: um it has opt-in uh devices in application, direct mode application direct mode. It means that you can shoot the persistent memory as their persist, persistent device. It means that you write the data into this device and you just shut down the machine and the data will not be lost, so I think it is a very good feature and and when you configure the persistent memory with the ad mode, there still can be two Motors to use this device. For example, the first usage is the fsdx mode.

A

It means that you format the persistent memory device with fsdx choice and the users get blocked devices. For example, you can see a device named AS, Slash, GB, slash P memory 0 in the host and the users can create a file system, especially kernel supported file system upon that. And secondly, second is the DX mode usage.

A

You format the persistent memory with DX choice and you get a chart devices, for example the slash DV slash DX 0.0, but you can still direct or operate on such devices and treat it treat it as block devices yeah and there. When you, when you use the fsdx mode on persistent memory, you need to use the DX choice to monitor device, then you can use the premium library with CPU to obtain the persistent feature provided by the pmm devices.

A

Unfortunately, if you use DSA to offload the operating operation example system memory- research configuration it will not always work because it needs the support from a Linux kernel, so I mean so it means that if we want to have a general DSA enabling work instead and not rely on the Kernel or we need to address this issue, so we raise raise the pr this PR 45609 uh half years ago and with this PR as it and this Pi is already emerging, itself's Master. We can leverage the system memory with DX mode directory.

A

So with this mode it can be very easy to use both CPU or DSC to to conduct the operations on the persistent memory device and it is equal and you don't need to uh change your kernel. So we think that this Pi is needed. So so at that time I mean that uh the the maintenance was uh he. He just revealed the sphere and merged that.

A

So so let me introduce the DML Library, so what is DML Library I think he has DML library is an open source Library uh to provide the data, manipulation on both CPUs and the DSs.

A

It can leverage DS's work, use exported by a kernel and the off-road is a the operations by CPUs and and when there are DSC devices available on the platform, you can use the ID config to create DSC work use and when DML library is used in the application, it will implicitly use the album to open idxd library and leverage create the DSA work use in Hardware paths and the improves those configuration Doc Page. We already share how to use IDE config to configure DSA working queues.

A

If you already enable yourself uh to offload a off-road persistent memory operations with DSA.

A

So we already introduced our work with the sync mode supporting in in Brewster, and the code is already available and we still have the plan to enable DSA in a sync mode, and we summarize the work uh into the following two steps as I described previously.

A

First, we need to change the current usage mode in premium module I mean the pmem device to specify to create help thread like the approaches in kernel device or sbq and Main device solution to to to conduct the operations on persistent memory by CPU in a single mode and adding here uh we have. Some colleagues uh is already on this uh task force and the second is that, then we will replace those I think operations conducted by CPU uh by DLC. We will Leverage The async apis in DML, Library yeah and the phase two.

A

uh We want to still offload a persistent memory or memory operations in C store, because system uses c-star framework and we will expect a good, a good thing behavior in system. So we will do the following works when this system is stable, for example, we would like to use the AC to to offload some other memory related operations. For example, the due cost copy or CRC uh related are checks and operations, and we still want to use DSA to continue optimizing.

A

Those operations on persistent memory in system or the CX attached memory, with the sync applicator in the near future,.

A

So the conclusion is that uh let me uh just uh review what I presented today will realize the overhead foreign to conduct data operations between persistent memory or memories while performing stores your workload, the the main or the loot cost, is that the data operations are between. Persistent memory is different from data operations of Gene membrane device, because for a little case there is RDMA or dma capability on those devices. So there will not be too much pressure on the CPU and especially when we deploy itself in hype converging influx structure.

A

If we equipped with the.

A

I think that the CPU will not be the I mean the main bottleneck, but I mean that if you replaced with the persistent memories, I think that you need to re-deploy.

A

And recalculate the the CPU resource to usage to use. So it means that to address this program, we propose to leverage the hardware based offloading engines, for example inters, iot or DSA, but but I mean that you can also use the other offering devices like the fpgs, yeah and they're. In my presentation today, we introduced the work to leverage DML, Library, 12, 12, Road, persistent memory by DSA itself.

A

We just introduced the work in in Brewster with already completed work in Blue store and the the ongoing work I mean that to use a sync libraries Pro provided by DML and to changes their uh the pmem device code in Blue, Store and I mean the next step is that we will continue uh uh this work.

A

We we will continue improving the performance on persistent memory or cexe attached memory through those uh memory of loading engines based on the essay or the potential I mean the other offline devices, I think for the performance report we may publish it later after SPI is launched. You know that uh this machine is still uh not ready in the market, so I think that it's all my presentation today, yeah thanks for your attention.