National Energy Research Scientific Computing Center (NERSC) Data Seminars Series, 17 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-05-17 - Nan Ding - Memory Disaggregation: Potentials and Pitfalls

Description

NERSC Data Seminars Series: https://github.com/NERSC/data-seminars

Title:
Memory Disaggregation: Potentials and Pitfalls

Speaker:
Nan Ding, Performance and Algorithms Group, Computer Science Department, Lawrence Berkeley National Laboratory

Abstract:
Memory usage imbalance has been consistently observed in many data centers. This has sparked interest in memory disaggregation, which allows applications to use all available memory across an entire data center instead of being confined to the memory of a single server. In the talk, I'll present the design space and implementation for building a disaggregated memory system. I'll then discuss the critical metrics for applications to benefit from memory disaggregation.

Bio:
Nan Ding is a Research Scientist in the Performance and Algorithms group of the Computer Science Department at Lawrence Berkeley National Laboratory. Her research interests include high-performance computing, performance modeling, and auto-tuning. Nan received her Ph.D. in computer science from Tsinghua University, Beijing, China in 2018.

Host of Seminar:
Hai Ah Nam, Advanced Technologies Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory

A

So welcome everyone to the nurse data science uh seminar series. um We're really excited today to have one of our colleagues from uh crd to share with us um a talk so uh on uh memory, disaggregation, the potentials and pitfalls.

A

Nan is a research scientist in the performance and algorithms group of crd at lbl. Her research interests include high performance, computing performance, modeling and auto tuning nan received her phd in computer science from chingua university in china in 2018, and we're very excited to hear about a hot topic on memory disaggregation that many of us are contemplating uh and trying to get our heads. You know heads around so with that I'm gonna give you the floor. Man.

A

Oh you gotta unmute. Oh sorry,.

B

Thank you, uh so um I think I'm going to turn off my video. So that way I won't be cut off, but not network. um Okay.

B

So uh today I'm going to talk about the potentials and pitfalls of the memory disaggregation, um so the outline for today I I want to uh have an introduction about what is memory disaggregation and then we go through some need to knows for descending a disaggregated memory system, and then um we can explore the design space and their implications.

B

So um first um I mean let's go through today and today's um node architecture and memory disaggregation. So we all know that compute nodes are the basic unit of today's hpc system. So you see compute node and memory resources are tightly coupled in each node, and users request the resources in a unit of a node.

B

So, conversely, memory disaggregation aims to decouple the compute and memory resources. As such, you will have a memory pool which has many memory only node, and these memory nodes are shared across the compute node. So such disaggregation allows for independent allocations of these resources, regardless, where a job is placed, in other words, applications could use all the memory available across an entire data center, instead of being confined to the memory of a single server.

B

So the reason that we want to explore this aggregated memory is because we see it's a common that in today's data center um memory, pressure varies greatly among jobs, for example, a nurse half the workload does not experience significant memory pressure, but 15 percent of the workloads use over 75 percent of the available memory per node. So um when planning for the next machine, we do not want to over provision every node with the maximum requirement, because it's a waste of resource, but also uh costly.

B

So today's solution is to equip uh with some large memory node. So the main issue here is that such solution is lack of flexibility. uh You can think that how do you foresee? What kind of workload is gonna run? I mean a nurse in the future and what? If the memory uh you provide today is not enough- or I mean too many so therefore, memory disaggregation is one way to go and there's also many research papers that recently published in recent three years uh in resource uh desegregation.

B

It all shows that uh memory design, this aggregation can address memory, imbalance and improve memory, utilization in data center and, of course, uh of course, cloud, um but we still have challenges.

B

So if you see this figure, it shows the ratio of growing flows and memory, uh bandwidth and memory, bandwidth and latency, and also the ratio of growing flows uh and network bandwidth and latency over time. So you can immediately observe that we have bountiful flows but lagging bandwidth and latency.

B

So actually the issue, just uh I mean arise from here- I mean, if everything improved at the same time, then nothing really changed, but this research tells us the processor will be increasingly imbalanced with respect to data motion. So, if we're building a, if we build a disaggregated memory system, it is likely to be of an increasing concern, also a challenge that the overall memory utilization on the system improves. But individual jobs performance may degrade with increased memory access over the network.

B

uh The good thing that we still have opportunities, because there's many emerging technologies come up like compute expressing like ddr5, which provide much higher bandwidth and much higher capacity.

B

So, let's give a quick look about what we have today and going to have in the future, and then we can, I mean better reason: the performance of this aggregated memory system. So at today's system let's say parameter we have a hbm2 and ddr4 and each node will connect with pcie for nic and then uh in the year 2024 frame we're going to have hbm3 but also ddr4, but here ddr4 I would assume it uh it build with 16 dimes.

B

So it has a larger um capacity and higher bandwidth than what we have today on perimeter and in year, 2026 we're going to have hbm3 and ddr5 and also pcie 6 nick, but I mean eventually the pcie nick you see, the bandwidth is much lower than ddr4 and hbm and hbm memory, so the network would be the bottleneck.

B

So I'm going to play with these uh emerging technologies to reason the benefit or.

B

Paid force of using a memory pool so before we really talk the details about this system. Architecture design, let's go through some need to know. First, is little's law. It is a very useful rule because it provides a simple way to measure the effect of a system. So let me give you a simple escalators example: let's say the parameters here are: uh if we have one person per step on this escalator and uh a step arrives every two seconds, and that gives the bandwidth of 0.5 percent per second.

B

If this escalator is 20 steps tall, that gives the latency about 40 40 seconds.

B

So uh if we, if we have one person in flight, that means that the achieved bandwidth of this escalator is about one over forty percent per second and then, if we want to saturate the escalators bandwidth, we're gonna need one person arriving every two seconds, and that means we will need 20 people in flight.

B

So you see the beauty of this littlest law. Is that it's tri-directional? So, given any two of these parameters, I mean it's either latency or bandwidth or throughput. We can calculate the third, so the takeaway idea of this little's law is that it takes latency time unit for the first person to arrive and we need bandwidth persons to get on the escalator every time unit.

B

So the same idea applies to the system: architecture, high level speaking bandwidth gives you performance. While latency is a constant overhead, you pay, but it can be heightened. So that's revisited the system, architecture uh figures so prominent her parameter. Node is a fairly standard, node structure. It has one cpu that connected to all four gpus and a node local ddr and then a all-flash launcher file system. But by memory this aggregation, uh the node local ddr- will be separated from the node resource and be a remote memory node.

B

So each compute node here I'm assuming uh we're gonna use an apu which combines a cpu and the gpu on a single silicon die and the both cpu and gpu share command bus to data common pass to the remote memory.

B

So um compute nodes need to go over the network, go over the network for the remote memory access. So it is natural that you know this aggregated system. You can access much more ddr memory than today, but now the question is: um how do you want to build such a memory node, and could you get benefit from memory disaggregation?

B

So this related to my next three topics: uh the first I'm I'm going to talk about the system design. Is that how many compute node and how many memory nodes you want to buy and then, um based on this system, architecture design, I will talk about uh how applications gonna uh gonna be benefit and what's the performance penalty of those workloads and then in the end, I will talk about a implementation of this memory pool.

B

So let's go through. The first topic is the system design? The fundamental question here is that if you have 10k compute node how many memory, how many memory nodes do you want to buy so here we're still assuming that one compute node is one apu and equipped with one nic and one memory. Node here, that's assumed is a ddr5 memory which gives you four terabytes per memory, node capacity and also equipped with one uh one pcie nic.

B

So, let's start from us from a simple case: if you compute node and memory node, I mean are in one to one ratio. That means by average each compute node can access. One memory knows capacity and then utilize all the make bandwidth for remote memory.

B

Another case is that if you are in two to one visual, that means you have every two compute node that access one memory nodes. So the average each compute node can access fifty percent of one memory, node's capacity and utilize. Fifty percent of uh remote memory bandwidth so you're going to have another 50 percent of bandwidth unused.

B

So the same thing applies to one to two ratio. That means one compute node can access two memory nodes by average, so per compute node. You will have 200 of memory, node capacity and use all the available bandwidths for this remote memory.

B

uh Now, um based on these methods, let's put in some real numbers, so the real numbers that are, you have 10k compute node, and then we can vary the memory node from 5k 10k to 20k, which corresponds to this two to one one, two one and one to two uh ratio case. There's three cases with the total remote memory capacity vary from 20 pack bars to 40 and then to 80..

B

So we can see this. um Let's see this figure first, so this figure, the horizontal, is the number of compute node and the number of memory nodes. If all the compute nodes that are required in remote memory. This corresponds to the three cases we just talked about and the memory capacity each node can get is like 2 terabyte, which is 50 of the node memory capacity and four terabyte and eight terabyte.

B

um So as we go from left to right, we are reducing the memory contention. So similarly, if we reduce the percentage of compute node that requiring remote memory, let's say only 50 of the nodes that are requiring remote memory, um each node, uh the the memory capacity each compute node- can get it then doubled. And then, let's say, if only one percent of compute node are requiring remote memory. The memory capacity each compute node can get is hundred fold. So, as we go down this heat map, we are again reducing the contention.

B

So this is for the memory capacity. We can then use the same method to have another heat map plot for the knit uh for the pci innate bandwidth we can achieve so again.

B

um This is refers to the three cases we just talked about, and here let me use pci 5nake as an example.

B

So if all the compute nodes are requiring remote memory, each compute node can get a half peak, half naked bandwidth in two to one ratio case and then reach the peak in one to one and one to two ratio case. Similarly, if the percentage of compute node that are requiring remote memory decreases, we are then reducing the contention and thus each compute node can get higher bandwidth until it reaches the peak.

B

So, following this method, we can have a bigger heat map. That's described the available remote memory capacity and available remote memory, bandwidth with pcie 5 and pc pcie, 5, nic and pcie 6 nick.

B

So, for example, um but I mean not, all of them are reasonable when we consider how we're going to configure the system in the future. So, for example, if we think it in the year 2025 frame hbm3 could uh give us uh 512 gigabytes capacity. So you don't want to buy a compute node that whose remote memory is smaller than the local memory that so all the blocks that are in this uh upper left region actually are not, maybe not a good choice.

B

We then need to see uh where future nurse gaps gonna land to decide which configuration makes more sense in the rest of the heat map. But if you recall at today's uh nurse 50 workload use 70 memory per node, so that means, um let's give a sense that maybe all the highlighted area would be a more tractable design space. That's about 10 to 20 compute node that are requiring remote memory.

B

uh So next I'm gonna to um to give a showcase that if we do a storage replacements for ai training, then how much benefit we can get. So here, I'm using a cosmo flow training uh benchmark uh to to to to do the showcase, so cosmo flow is a scientific machine learning benchmark in our perf hpc training, benchmark suit. It uses a 3d conventional neural network, with five convolutional layer and three fully connected layers.

B

uh So all the data here I use refers to uh pn uh refers to the ps pmbs paper listed here, but but you know the data that matters is that for cosm. For this cosmopolitan uh case we need the training set is about five terabytes and the flows per sample bed is about 15k. So that's the.

B

How how this custom flow case can get can be benefit from a disaggregate uh system. So, first we need to. uh I mean to pick up a system configuration so again, if you recall that the 10 to 20 is a more tractable system configuration based on what nurse happened today. So again, if we ever pick another system architecture, let's say we have 10k compute, node and equipped with 1k memory node with further narrow down our system configuration to to the to the blocks.

B

Here I mean the overlapped highlighted blocks, so it gives the four terabyte memory capacity per compute node and correspondingly you can reach the snake peak. If it's pcie 5 you're going to reach 50 gigabytes, if a pcie 6nick you can reach 100 gigabytes so in the cosmo flow here in the cosmoplog case. So today you can either do it. I mean in a larger file system. You, you just read, load all the data from luster, and that gives you coherence, but you have no performance.

B

Alternatively, you can also distribute all your data in your no local ssd. It gives you better performance, but you you don't have coherent. You still have to pay expensive data synchronization.

B

So in a disaggregated memory system, let's say we take uh again: we pay a 10k, compute node and only 10 compute nodes that are requiring remote memory. um Let's see how how much better we can get.

B

So in the cosmopolar case, if you recall, we have five terabytes training set and we need 15k flops per sample bed. So again, if we fit in this disaggregated memory system, we can in this in the in the pickup system architecture. We can definitely reach the network peak.

B

So that tells us with a peak pcie 5 nick we're gonna require uh 770 teraflops compute capability to finish this cosmo flow work, so fspcie6 we're going to need more, but I'm going to give you a reference about how how much computer capability we can provide at today's and future gpu, for example, on a100 gpu.

B

It can provide uh 218 tara flops by uh ip16 tensor, which is required by the cosmoflow. uh This number actually is much lower than what we're going to require by using pcie 5 nick. So if it's hardware, gpu hopper gpu, is a newly announced gpu this year by nvidia, uh it can provide 700 tiraflops, which is also slightly lower than what we're gonna require by pcie five nick and also much lower by pcie six nick.

B

So you can see that by using a disaggregated memory system, it shifts the bottleneck from I o to compute, which is good, because if you recall the figure that I showed at the beginning of this talk, slopes always grows much faster than bandwidth and latency.

B

So this is the first case that we see how ai training workload can be benefited from memory disaggregation, um but I mean we have. We of course have various kind of applications. So next I'm gonna talk about what kind of application would be benefit and what will be the performance, bottleneck and performance penalty.

B

So again, let's look at this memory capacity and the bandwidth figure. So this is what we have today. We have hbm2 as local memory and we have pcie for nic and then in year 2024 we're going to have hbm3 and pcie 5 nick and in year 2026 is still hbm3 and pcie 6 nick you see, there's a big cap between the local memory and and network bandwidth. So again, the network bandwidth will definitely be your bottleneck.

B

So the general question is that how will memory desegregation change our way to handle data, and do you have a chance to operate at hbm speed and will not be bounded by the pci nic? So here I'm using a memory roof line to present this uh capability?

B

So let's do a visit of the traditional reflect while the traditional float roof line. Tells you which takes a longer as either data movement or compute. So if you dot fall into four into the left part of this wall, that means you're, gonna, be memory bound and data movement takes longer. If you, if you don't fall into the right of the started wall, that means you compute your compute bond and your compute takes longer so memory roofline, um I mean take the similar plot, but it shows a very different insight.

B

It might have tells you whether it's your local memory takes longer or remote memory takes longer. So in this plot, so the horizontal is a local remote memory ratio and the vertical is attainable gigabytes per second. So correspondingly, your peak is not flows, but your peak is a local memory bandwidth.

B

So if you plot, if you have a dot that fall into the started wall to the left of the started wall, that means you will be bombed by the remote memory and you, you remote that so, in other words, as your remote memory takes longer, so if your dot fall into the right of this dotted wall, that means local memory bond and local memory takes longer. So, let's see how where we can get.

B

This is a plot that reflects what we have today. So today we have pci fornic and hbm2. That means we're. Gonna have to write for hbm memory, 60 uh two times then pcie four nick and then we can operate at at hbm2 speed.

B

Otherwise, you're gonna fall into the left of this dotted ball, which means your remote memory bond and you will eventually, by bond by the pci nick so in year, 2024 when we have pcie 5, nic and hbm3 you're, going to refer more data from hbm memory, so that you can operate at hbm3 up uh speed in year 2026, since we have a much faster nick, it pushes the dotted wall to the left. You can refer. I mean the same number roughly.

B

The same number as today is about 65 hpm memory than the pcie neck and you can operate at hbm3.

B

So the main takeaway from this figure is that if you have a faster make that will push the dotted wall to the left and you will spare less effort to refer the local memory and to to overcome the neck bandwidth.

B

So, let's see how applications goes from here, so this is a um so this is. um This is plot in this plot. The horizontal is the memory capacity and the vertical is the local remote memory ratio. So if you recall in the year, 20 26 frame hbm3 would give you 512 gigabytes uh capacity, so anything that we would be smaller than uh 512 gigabytes would fit in the hbm in year, 2026 and and then for the for the orange zone that refers.

B

You can be benefit from memory disk aggregation because you have a higher local, remote memory visual, so you will operate at hbm speed. So, alternatively, if you fall into this green zone, which means you have large capacity, but you have low local remote memory ratio say smaller than 65 and you will. That means you'll, be you will be bound by pcie nick so here, just to give you a reference about how uh how much we have at today's parameter, node, so you'll, see hbm's ray in year.

B

2026 will give you much more capacity than today's parameter, no local ddr.

B

So that's the let's see the first, let's see the ai training, uh so you can think in ai training workloads.

B

um Local memory refers as your activation layer and remote memory refers to your sample mini bench, so we just show the cosmo flow, but also the deep cam and both uh cosmo flow both fall into the orange region, because you have large capacity which cannot fit in hbm.

B

You also have a high local remote memory visual, but uh if I think another case is resnet, which has a relatively low memory requirement, so in the future you are likely to fit all your model into the hbm memory, but I mean you could also have some other special training model which have a very sparse activation layer.

B

That means you will fall into a screen zone because you have a low local, remote memory ratio but generally speaking, model size, ai training model size grows expandedly, so at the average, all ai model increased 25 times every two year and training sites only get bigger so which so in general, and moves by air training workload is moved to the upper right and into the orange zone. So that means memory. Disaggregation could benefit in general benefit ai training workload, but we also have many other workload, for example the graph analysis apps.

B

uh You can think if you have a small graph, you are likely to fit in hbm and then actually you would you wouldn't care this dissertated memory system. But if you have a large graph, because this binatural such graph analysis application, would have a low local remote memory ratio, so you're going to eventually be limited by the pci nic and then for the hpc applications. I would say it may touch all three zones but depending on the memory patterns, so you see, uh we do have many applications that could be benefit uh in in this.

B

In search of this memory, that's agreed uh uh in such a disagreed memory system so now. The next question is that if we see the benefit and how would you build, how do you implemented such a um memory memory pool? So here, let's uh review the latest law. So the latest law tells you that it takes latency times unit for the first person to arrive and we need bandwidth persons to get on the escalator every unit.

B

So remember that to to maximum the escalators bandwidth, we're going to need 20 people which equals to the latency multiplied by the bandwidth in flight, so that we can maximum the escalators bandwidth. So the similar thing applies to the system architecture.

B

So here we're going to use this little law to calculate how many data you need to keep in flight uh so that you can hide network latency and do you have enough parallelism? Do you have enough concurrency to hide to support such number of data to be in flight? So, let's consider uh in a network attached this aggregated memory system.

B

We consider the network latency as about 2 microseconds, so in pcie, five nick we're gonna have 0.12 megabytes data in flight to high latency and in pcie six nick we're gonna have we're gonna double this data.

B

So let's see how let's see if we have enough uh parallelism or concurrency to hide latency uh again here we're gonna use a concurrency roofline to to tell you that. Can you hide pci latency, uh let's revisit this float, you fly again, so the float roofline tells which takes longer is either the data movement or the compute, but the concurrency roofline tells you whether you can hide uh make latency. So in this plot it still looks very similar with the traditional traditional reflect, but tells a very different insight.

B

So this figure, the vertical- is a sustained gigabytes per second and horizontal. Is you can think it as a parallelizer? So you can think it as a hardware thread number of hardware threads that could be could be scheduled in flight and then correspondingly, your uh sittings here as a picnic bandwidth and the the diagonal. The diagonal, the diagonal blue line is the number of bytes you need to keep in flight per thread. So if you fall into the left of this dotted law, a dotted wall, that means that that means for this kind of solution.

B

It's hard to hide, pci latency, you can. You cannot reach the uh pcie naked bandwidth, but if you are fall into if you fall into the right of the started wall, that means yes by using this solution, it's easy to hide latency.

B

So that's, let's take a look uh at the two solutions here so here I want to talk about two solutions: it's either the hardware solution and software solution. So by hardware solution.

B

That means we can liberate the hardware threads for data movement and each thread will handle one cache line, which is a 32 byte and by software solution. That means as performance by the os paging. We can vary the page size from 4k to 1 megabytes, to 4 to 2 megabytes. So, let's see how it works.

B

Let's see the hardware solution first, so because hardware solution is deliberate. Is the hardware service to do the data motion, so the number of load store cores in a gpu or say a apu um determines how far we can go. So, let's review how many load and store cores in the a100 gpu and the latest uh the newly announced hopper gpu, so on a100 gpu, you will have four thousands those store cores.

B

That means uh you can you can schedule four certain stress to for for those four certain course to keep data in flight and on helper gpu. You have 4.5 k series to operate on those cores. So if you see this figure here, each thread could access one cache, one cache line which is 32 byte, even though you keep all the load store cores busy. Your schedule authorized to keep the load store core pc, you can you can barely reach the pcie 5 uh peak, but you can never reach pcie 6 nick.

B

So generally speaking, pure hardware solution cannot meet the requirements in year 20 25 frame, but, alternatively, you can use a os paging, which is a software solution. uh All the purple lens here refers to the os solution and different uh diagonal lines. Here refers to different page size, so apparently page size, 4k cannot work, but if you have one megabyte or two megabyte page size, you see here you can easily utilize the pcie 6 bandwidth and height latency.

B

So from this analysis we can see os solution can easily saturate negative bandwidth with paste size larger than 256 kilobytes. So os paging would be a more tractable solution for the disaggregated memory system uh so and another. In the end. I want to talk about by implementing such a uh disaggregated memory system. How would that affect programmers?

B

So how do you program on this uh disaggregated system? So you can think that by either hardware solution or software solution, you can we can provide inflated interface or explicit interface. So interplay is it implicit interface, which means that um it's program friendly, you can think you just a program like unified memory, but it is generally expensive.

B

Alternatively, you can. We can also use implicit interface in a software solution, which means you can uh you can think you could prop program on this memory pool using like cuda map copy, which is cheap, also, but also program friendly.

B

So, alternatively, all the explicit interface, for example, like schmann, is a program unfriendly, because you have to rewrite your code using, let's say model, to keep your performance and also makes, and also you you have. You need to spare more efforts to make your code run correctly, um but, generally speaking, software solution is relatively cheap than hardware solution.

B

So, in the end, uh just as a summary of my talk is that system architecture design needs to be determined by both data center and applications.

B

Remember in this heat map, data centers have the budget to determine how many memory nodes you want to buy, but applications running at that data center determines determines the occupancy, so it requires both both rows to determine which system architecture configuration will be more reasonable, and then we estimate the benefit or say no benefits of different kind of applications.

B

Let's say we should look at local, remote memory, visual and memory capacity requirement instead of using applica applicable scenarios, because local remote memory ratio really determines whether you will be uh bonded by the naked bandwidth or latency, and that gives you a more precise performance panel penalty for your applications and at the end we talked about the two uh solutions of this aggregated bandwidth system.

B

um The conclusion is that the os paging is a more tractable implementation for the memory pool compared to using hardware sweat, but we still have some future work that needs to be done, for example. uh Currently, this is all theoretically analysis of whether we could be benefit or not benefit by the memory system. We could look also look for some other uh solutions, like hardware stimulation using ferrocene, and we also need to evaluate uh various applications that to see if there's more critical, matrix uh need to be considered. We als since this memory.

B

This algorithm is this. This aggregated memory system is actually all stacks question, so we need to collaborate with all other stacks to see um what is missing or what uh what what would be the potential.

B

So that is all for my talk.