Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Large-Scale Practice of Persistent Memory in Alibaba Cloud - Junbao Kan & Qingcan Wang, Alibaba

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Large-Scale Practice of Persistent Memory in Alibaba Cloud - Junbao Kan & Qingcan Wang, Alibaba

Persistent memory allows programs to access data as memory, directly byte-addressable, while the contents are non-volatile, preserved across power cycles. Alibaba have millions of databases and memory prefered applications which need massive of memory resource and make a huge cost every year. PMEM device provides the high performance and lower price which have been widely used in Alibaba Cloud. We have developed a combined system focus on PMEM resource optimization and capacity scheduler, which is widely used in our Kubernetes platforms. With the system, PMEM device is used as kuberentes volume object, and can be configed in different types: kmem, quotapath, lvm, direct. Also we optimize the scheduler to implement that PMEM device has best match with numa node.

A

Hello: everyone, my name, is from alibaba cloud today, my colleague jimbo, and I will give our talk about the large skill practice of persistent memory in alibaba cloud on kubernetes.

A

This is our profile. You can contact us through github or email. This is our. This is the agenda of today's topic. In the first part, I will introduce the system architecture of pmm stack.

A

The second part we will introduce the practice of pmount stack used in our company and the third is the demo and the fourth is the other related works about pima.

A

So why is persistence? Memory presence memory is a higher performance by two addressable memory device being on the memory. Bus, low emp memory has the memory. The ram like access through data and persistence memory has a following filters.

A

The first one is data, persistence and the second, the the second is the higher performance and the light capacity and the low price, and why we used the p memory p map in our company.

A

It is the first day's cache database is large capacity memory space and the second is large. We want to save our cost.

A

And then, let's see the necessary of using p mapping, kubernetes pmap is typically used as field system volume, 2 application, which can be provided as a persistence volume with csi first containerized, more application. I containerized in currently.

A

Many payload eye deployment on kubernetes platform and use the storage results as volumes, so pmap used in container becomes more and more requirements.

A

Secondly, automatic kubernetes manage pmap device automatic with kubernetes customized controller pmap can be initialized as defined policy, and also pmom device can be provided as namespace and a field system automatic.

A

Thirdly, pmap can provide more than tb capacity on onenote, so resource sharing is necessary between different users, so the resource limit is requirements in the production.

A

And then is capacity aware capacity on a single node is limited, so we need to report resource capacity of the node to the scheduler. Scheduler will determine the location of the new port based on the capacity of the pmap in the entire cluster.

A

This is how this is the architecture of pmom stack.

A

That's the dark with the architecture diagram on the right, as the bottom of the graph is node resource manager, node resource manager is responsible for automatic initialized pmap device, and it also formats the device and mounted as fuel system which can be used as directly by accessed by application and node resource manager is deployed as demo set and works on each node in the cluster.

A

It also reports the node resource capacity, which can be used by scheduler, then is a sas liar. As we mentioned, pmap can be used as storage to application, so we can manage it. Using csi plugin in kubernetes pmap will be used as persistence volume and can be managed by kubernetes volume system under csi component pmem is formatted and ext4 field system will project quota featuring able, and we also provide volume prevention, volume, amount, volume, capacity, resize and volume monitor full csi feature is implemented above csi liar.

A

We provide scheduler, plugin and auto resizer component with auto resizer. The pm volume can be extend according the capacity usage, the pmap volume can be extended. According the capacity usage and with schedule extender, we resize capacity awareness of p memory source at the top of the graph.

A

There are some crd, all config map, which I used to record capacity information or define p memory, resource topology and then last week, look at the detailed, inter implementation of the node resource manager, node resource manager returns on every note of the cluster and approaches the config map, which defines the pmap resource topology in node resource manager.

A

Please alter the pmap definition on this node. It will start to check pmap device and guarantees and then generate to the target.

A

Namespace node resource manager consider device as storage and format it to fill system application use pmap with project quotas, field system, so node resource manager will enable the quota filter on the eact4 field system and also enable the dax filter, node resource manager, records the node resource capacity to know the resource crd for each node. The node resource cr will be created for every node in the cluster.

A

If there are more than one project quota field system in onenote, they will be recalled in the same node resource year. Node resource topology defines a p memory source planning in each node. There is an example of the definition.

A

Name means the field system directory of the node and the key and operator and value is used to match your target node. If onenote is matched the key value, it will follow. The defined pmap policy topology define the resource planning, it defines the resource, type format of options and pmap regions with node resource manager. The pmem stack implements resource automatic and resource capacity reporting.

A

And then, let's introduce the pmount csi pmom stack implemented. The four csi filters for pmap type volume, unlike distributed field system p memory source, is the local resource on the node, so the prevision glitch operations will be executed through grpc volume, prevision csi controller will, it result is responsible for creating volume. It called grpc to the remote server on csi agent side volume. Prevention is only support in the lazy building, which sets the volume bounding. Multi adds with four first consumer volume.

A

Delete with jfc call csi controller will remove all the fields under the subpass, which is related to the pmap volume. If you don't want to remove the data, you can set the reclaim policy and retain volume resize.

A

If the volume capacity does not meet requirements, you will want to expand the capacity with pmem csi you just resize the pmac resizer pvc size and the system will expand the capacity automatic next.

A

Let me introduce the volume auto resize cyber system. We have introduced the volume online size in the pmm csi part, but the only online resize is not enough in the production cluster automatic resize one volume capacity when the current usage is in the emergency, so we develop the auto resize controller first introduce the ultra resize policy. It is designed as a crd, which defines the resize policy for the pvc.

A

It can be defined when which volume applies to a policy and when one volume need is to expand capacity. The policy supports a critical volume of size of the percentage.

A

Then let me introduce how to implement scheduling, to use pm more reasonable and efficiently. Our implementation is based on the scheduling framework. The scanning framework is a blackboard architecture for the kubernetes scheduler. It adds a new set of plugin apis to the existing schedule. The scheduling framework defines a few extension points.

A

At the same time, we have also implemented some strategies in the big data and ai through the scheduling framework, such as gun scheduling and the elastic quota of capacity scheduling. These functions are open source in the scheduling, scheduler plugin project on github welcome to try zone.

A

Then let me introduce the plugins for capacity scheduling. There is true selection policy in capacity scheduling. First, we allocate port requests to the same region based on capacity.

A

Second, when some, when we choose the region, we will use bin pack strategy to give the priority to the region which, with the lowest remaining amount so that they can meet a port with large resource requires in the future.

A

For capacity scheduling, we may implement to the following four extension points: filter filter out to the nodes that continue to the requirement of pmap score.

A

Select the optimal pmom scheduling result by algorithm reserve resolves the optimal pmap scheduling, result to prevent reallocation to other posts, and if failure occurs in the bonding cycle, it will clean the pmap scheduling, result: pre-bound update the pmap scheduling result to the annotation of the pvc, and then we will introduce the new amount of air scheduling, because cpus located in different numerals have different delays to to access the premium of different circuits.

A

So we will have the following scheduling: policies based on human node first try to allocate the cpu of the same newman node to reduce the switching of applications across newman nodes. Second, select: the combination of pmem and cpu, with shortest distance similar to capacity scattering. We also implement four extension points in human or their scheduling, filter, filter out a node that continues to a requirement of the port for cpu score. Select.

A

A combination of pmem of cpu and cpu, with the shortest distance result, also resolves the optimal cpu schedule result to prevent reallocation to other posts, and if failure occurs, it also will clean the result. The force is pre-bound and it will create supposed sig rope's request. This is a crd to store the cpu scheduling result and if you want to know more about the cpu scheduling, you can click the link below it will introduce more details.

A

So this is also.

A

Of the cloud native p mount stack introduction, the following content will be introduced by my colleges dream ball. Please welcome.

B

In the second part, I will introduce the practice of pm device using animal cloud.

B

Let's take a look at how that is used on processing memory stacks, as the left graph shows the processor memory device, information as dx file systems, and that is access memory device with direct io by mf system, calls for the right operation, radius rise data directly to dram and reduce also raised data to persistent memory. Until your operation retains successful for the for the grid operation, difference process, logic is applied to business data and metadata for metadata.

B

That is read from dram, and if there is no data, matched radius will read from past memory device and catch the data to dram and the host data for business data. That is, we will read the data directly from the personal device.

B

The right graph is the overall attacker as the bottom layer is the custom memory and parameter hardware resource provided by alibaba cloud. Our type of hardware is about cloud linux, os and the path layer is cognitive communities platform which provides a number of extensive plugins, such as audio sizer, scheduler, says plugin and resource manager.

B

On the top of the system, it is customized and radius engine and radius applications by running the business um stacks. The ready service achieves containerized deployment and automatic operations.

B

Here is the benefit of using this memory stack for the ready service in the old ready system because of the limitation of memory capacity already service can't miss the requirement of a live question cash system also because of the high cost of memory that is service, costly, high and in software architecture.

B

All system is using both cash and storage resources in the new system. Personal memory source can provide large radius instance, which is important in other database service scenarios.

B

We consulting a radius instance online and get the phone aesthetics in terms of performance compared to dram. Personal memory, achieves 90 percent performance and in terms of cost compared to dream. Personal memory achieves 70 percent cost, so the memory source gets a good performance and cost benefits and for software etc. In the old system, the architecture has three layers will contain applications they really sketch and possess volume. In the new system.

B

The is at the alexa has only two layers which contains applications and radius cluster no need for some volume, then to see the radius cluster itself plays both sketching and data persistence.

B

The third part is a demo.

B

First, we will, we should create a kubernetes cluster from our cloud hq service and add a node which contains the contains nodes we can see. This class has four nodes and oneness is memory type.

B

Okay, we can log log in the p memory.

B

B

And find the amount of information by the for the pmap. You can see, there's no path mounted by the pmm device.

B

And the premium is on the tv path.

B

There's no reason: there's two namespace pmmd device.

B

And step 2, we will install the new resource manager.

B

And check the notes, just mentor product is running.

B

Okay, our product is running and then we will apply. The traffic map check the command definition.

B

The customer defines the target node as the color path type, which means the device will format as ext4.

B

A system with product called ha enable and is mounted to the path of mnt and pass lines and the key operator and the value define the tax node, which will follow this policy.

B

Now we can check the tax path is monitored to the small demands on node.

B

Okay, the pmi device is mounted by the path mnt pass. One.

B

Okay, the step three: we will check access, plugin, install and install the third class.

B

The sales plugin is installed and the port is running and we will apply. uh We will check the third class as well class, defines the volume type as code path, which is same as the concept map and the load path as mnt password, which is same as custom f2 verify the third class.

B

B

We create a service application with the premium volume. Yes, so it says.

B

You can see the solar class name is defined as alibaba ali cloud pm quota, which is same as we created right now.

B

Now we check the port, which is quitting and it's running now we check the pvc. The pvc is created right now and the pv is also created and we log in the port.

B

And check the main point: we find the target path. Data is mounted by the persistent name person memory device and the size is 3gb, which is how expected?

B

Okay, we resize the pvc with the application online.

B

We change the style from 3g to 4g.

B

And we can find the pvc and pv size. Tv has changed from 3g to 4g, and the pvc is some slowly.

B

And last step we can and we can check the audio size controller features we can find. The auto size controller is installed in our class cluster.

B

The product is running.

B

Okay, the pvc has started finished from 3g to 4g and we can check the audio size policy.

B

And the pvc selector means it defines which pvc will be watched by this policy. For example, this policy will force the pvc which has a key app and value index code and the conditions define the volume conditions which can detect the actions and suppose the free size type and the procedures type. There is a three cell type. It means if the tree size is less than 2 gb, it will trigger the actions and the actions define the the volume upgrade actions.

B

It can contain more multi actions- and here is only contains y axis, and this axis defines the top as volume expands and it will expand the volume 2 gb every time and the next size is 20, gb temptation.

B

Okay, we apply the polysize policy okay and we and then the copy data to the code target for the target path continually and check the capacity chains.

B

It will copy the files to the target path and I will check the volume change volume, question change.

B

The capacity trader will going to be like this graph as the use capacity grows and the volume is continuously expanded.

B

Okay, this is a demo.

B

And the last type last part is the last part is some related works on the personal memory, the addition to be used as a storage device.

B

In addition to use views as a as a storage device, custom memory can be also be used directly as a memory device. We work on the memory mode of his.

A

Memory following.

B

Two scenarios first assembly used as a template ss as a graph source, the form with measure the person memory device as ev ex type and add it as a new node. Then csr plugin will monitor this memory. You might note, as a local volume, by temp access and provide a two application with pvc and tv.

B

The terminal fs volume can be dynamic, provided and multiply csr plugins the communities can provide the whole life circle, management for the terminal, 5 volume and it it could be more monitored by promised and resized by all sides controller.

B

In this scenario, the personality device is used as high performance, local storage. Secondly, personally can be used as memory case, some same as above scenario, to match the person memory device to dvdx type and add it as the newer nodes. You can run the online applications on the ram and if, if the application begins to stop the system, can move the memory data from offline application to personal memory, when the offline application will collapse, the system can impose the main data from host memory to dram immediately and style the application.

B

In this scenario, this memory device is used as a memory to catch the breaking data and split up the applications. Up and last I will introduce the field works related to related with the personal device. We will look forward to find more business scenarios to customer device. For example, in the biggest area we will do more effort on community source scheduling, to enhance the device performance and make the source path between key mem drafts and cpu in the field of memory.

B

Poor, the person memory can be used as a chip memory source, and we expect expect to implement the memory dynamic provision in cognitive environments. Then we can attach memory instance whenever the product is running.

B

That's all my topics. Thank you very much.