Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SPDK Based User Space NVMe Over TCP Transport Solution - Ziye Yang, Intel

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

SPDK Based User Space NVMe Over TCP Transport Solution - Ziye Yang, Intel

Recently, NVM express releases the new spec of TCP transport (TP8000) for NVMe over fabrics. In this talk, we would like to introduce the design, implementation and development plan of TCP transport for NVMe-oF in SPDK . Currently, SPDK implements both TCP transport in host and target side, and it can be tested against Linux kernel solution with good interoperability. Besides, some experiments results will be presented to demonstrate the performance and scalability of SPDK's NVMe-oF TCP transport implementation. Moreover, we will introduce some techniques for the further performance improvement of SPDK's solution, e.g., (1) leveraging user space TCP stack (e.g., VPP + DPDK) to replace the kernel TCP stack; (2) leveraging some features of hardware such as ADQ on Intel's E810 NIC.

https://sched.co/NruW

A

B

Good afternoon, so the topic today is sdk-based in userspace, any vimi vimi are over TCP transmission, efficient solution. It's basically.

A

B

Ston that you know the fabrics.

A

Agreement very well so.

B

This is the agenda today, and this is basically introduced as PDK. Anything me out: development, history and stutters and basically we will talk about SDK, TCP transport, introduction and then conclusion.

B

For start, the SPT can be of target online before that we will talk about why we any fabric agreement well previously in the machine will use this kashi agreement to compile a drive SSD the elite. That's the case for the upper layer.

A

B

We are exposing this fast device to the.

A

Internet, if we're.

B

Using the pistacia protocol than PCIe SSD emerge and then we have this and many protocol.

A

B

A

B

Migrates one any disk and allows remote visiting accessing. We can still use other protocol, but there's the risk of overhead, for example, and many critical have to be changed to a Stasi protocol and transported to target site and Astarte protocol would have to be on our system change to a Stasi and a many. So this overhead is pretty large, pretty consuming. So.

A

B

Have PCIe SSD Hardware.

A

B

Change from star C agreement to any protocol in order to better serve the remote access from a.

C

Better service for.

A

B

Stasi protocol I mean your old fabric.

A

B

This is the history: I mean your fabric. Protocol is pretty young from 2016 onwards. It's doing solving.

B

The murders kernel.

A

B

Sdk kernel and SDK are two.

A

Different solutions.

B

Interoperable so.

A

B

Is the history of an SPD k, anyb target timelines from July 2016 to 17.

B

March to July, we have functional heartening and we have done some kernel tests and from November to 17 to 18. We are serving the our DMA transport improvements. We are still doing it and in January this year we released the tcp transport support and from April this year onwards. We are.

A

B

With our optimization first for a many fabric, protocol standards has been renewal according to the spec and cold and also in order to be more compatible with kernel. We are improving and doing tests regarding interoperability and our performance and based on SDK diagnosis.

B

We need to there are improvements in this page shows that this is the SDK enough host tan line or initiator development process and I won't go into great details here. It's similar to the target side.

B

That bricks target.

B

Code design- and these are the details- these are the.

A

Details that we used.

B

Try to prove that based on SDK, we can have the menu fabrics. Target performance is guaranteed. First is important. The first point is important. As PDK target side I used.

B

A

M&Amp;E driver.

B

A

B

Advantages that tells you the application if they want to visit the MST.

A

B

The writing and reading disks will have to go through the system, scheduling and will have to go through the definition system and, for example, the first device. And then we go to the I/o stack. It's pretty lengthy and.

B

And we had to switch from have to switch the contacts between the clients or the customers and also for a colonel. We have several lock in IO stack, so there is a heavy competition in regarding the resources, so the performance is not very well and if we're using the NBA to drive the whole process that we can solve this problem, for example, we can reduce.

A

B

Contact switch or the times needed, an IO stack will be reduced. A number will be reduced greatly.

A

B

Implementation.

B

Will not be boxed by a competition of resources in forum client-side. If you create an IO QPR, each QPR will have to be divided into Commission queue and exhibition queue and for.

A

B

Thread can control the self and it can control the I/o qpl, that's created by itself, and each thread are controlling the I'll copy are created by itself. Another submission and completion.

B

Will not have any problems regarding the resources that were not computer resources, and one of the precondition is that you will use the linux uio to drive it. You can draw a memory map.

B

Okay, it's doable and that's.

A

The first point.

B

A

B

Now verified by lots of the customers and it's being put in the production environment, and are you tested it second.

A

B

Not sufficient, we need to provide other solutions. We need to provide a framework for programming. This is called as PDK encapsulated, socket, API and premise target. We have a good polling idea in SDK target time on each CPU chord. We have a new thread, there's one and only one thread that is running or operating a one specific task. They will have a group polling function and this function is.

A

B

Focused on processing all the connections in the group and.

B

What connections that's being referred from the.

A

B

If we accept this connection, then we have will be rescheduled to one specific, severe core for processing. If you have only one CPU cordon, that means we have only one good.

A

If we initiate.

B

This fabric target with multiple CPU core than each CPU core will have its own thread. Then, if one connection is taken over by the calling group, then the polling group on each CPU will have to be operating and assigned a connection, not other connections, that's handled elsewhere.

B

So to a great extent, we will minimize the competition for resources, and how can we assign the connection on each polling group that will be determined by algorithm? Well, the current algorithm they were using is.

C

B

And the third is deducted from second point and.

A

B

Connection will be mapped to a me and VME io q and we guarantee, by doing so the cpu will operate and handle this connection.

C

B

There will be no competition for resources and the fourth point and many c'mon handling.

A

B

A

B

A

B

Deployment layout and there's.

A

B

To a great extent, we can improve its performance.

A

B

For io q depth.

B

So this is SDK MF target situation. This is the overall situation. Now, let's see SDK the transport.

C

B

And we can see that in SDK for transport we have a wrapper and in many fabrics protocol we have a definition for.

A

B

For example, we have fabric channel our DMA and TCP.

A

And currently, at.

B

The green parts.

B

Decide on the protocol and its journey developed by NAT, calm and Bob we're still reviewing it. You can still.

C

See the progress.

B

C

B

Ccp transport in SDK, we support to TCP realization. Currently, we based on the kernel, products and vex processing, the are two different categories. Why do we need to keep these two interfaces? The main reason.

A

Is that SDK, the customer.

B

Target is operating on their customer specific environment and, if you're.

A

B

On the kernel TCP, then you will have to occupy.

A

The service than the.

B

Many commands transported from this.

C

B

Going to the networking card than these and many clones.

A

B

Handling will not.

A

B

Kernel stack: if we have ATT be transposed from the customers, then the.

A

B

A

B

Handled by P and D.

A

B

Example, pulling mana driver and packet the packages and transformers and analyzed the head of the packet and analyzed focused on the M&E command.

A

B

Transported to the customer stack.

A

B

It back to the customer protocol.

A

B

P ND data and as a whole stack will guarantee that the customers can run through the whole process and the io yellow copy will also be realized. If the I/o is coming in through card, then we will assign it with a buffer and the buffer will be used in the whole implementation of the aisle.

A

Implementation.

B

So this is an idea of status and the first we will support the TCP transport. We have VPP integration and also still in development process and VPP integrations. Do.

A

B

Issues to handle if the VPP.

B

If the VPP, our customer protocol stack, is transferred to other platform, it won't be that step off, so we need to fix a lot of bugs the.

A

Standardization is doing progress not and.

B

Then embeddable embeddable library cannot be integrated with STS PDK. We have two processes, one is the AMF target process from SPT and the other is the VPP process, and if we have data coming in from VPP collecting all the packets and.

A

Processed by tcp.

B

Going through the memory process and shared to targets, and vice versa, so the performance is not very good, not ideal, so we have a wrapped TCP interface and the customers can use the SDK socket API wrapping to integrate with the other customers stack. For example, Tencent's have stack. These are options on the plates. This is, there are a lot of room for organization.

A

B

A

B

The service and performance.

A

B

Pretty much like the previous slides, so it's pretty much similar to the programming now have talked about the TCP transport performance of generation.

B

There's still, a lot of rooms are still a lot of issues that we can handle and also.

B

Sdk and enough, we still using the polygroup for all the socket management we.

A

B

A connection for the first time: if we stop the connection, we will put it in the epoch group and later we will monitor the socket all of the data in event. In Linux we have the equal links system cool and we will see you what are the active connections and then these are what we will do and the second is always mentioned for TCP connection. Optimizations we're still using the SDK wrapping capacity, integrate, make the integration and second.

A

B

Offloading library that provided by the company, for example, Matt Ross.

A

B

Have a Marek's message passing API, so it's also a socket interface and we can directly upload.

A

B

Word of CCP to its own network network card and there's a precondition of doing so. First, while using it or used a little preload to preload.

A

The library replacing.

B

The TCP socket from juleps second we'll need support of hardware for pub sides, both parties and third TC pp6 handling from it's pretty simple, so the spec has been determined. We have to check and all these situations. Finally, for Emily CCP request lifecycle management.

B

We use efficiency stack and we can still realized it in order to guarantee that the performance.

A

D

C

Whether you bring all right if.

D

It's a green command after the buffer has been allocated. It can just be given.

A

D

A

D

Read it in capsule video, then.

A

D

We received the data from the remote device. We can just write into.

A

A

D

We have to trigger the ready to.

A

D

A

D

This dip into theta, they were gifted.

A

D

And then to execute and.

A

D

C

D

This to go to the initial stage, so this is the whole process. The red dotted line means.

A

D

Is similar to the STD case.

A

D

Pretty clear, so this is helpful for you to read some coats.

A

D

We have the performance.

A

D

Consists of three machines: why is target.

A

D

Have a sixty any and we disks they can put it into long subsystem.

A

C

D

For the initial site, the initiator site.

A

D

Can have the one to six team connections and for two it can tell us as much as two.

A

Connections for this page, you.

D

Can see the IO scaling.

A

D

Divide the target site into four parts and the initiate her side. They can send sixteen connections, so we know that they are 32.

A

Connections for.

D

The left to wonder Cody is one and the food a left or right or on the Qt is that okay.

A

D

The line is the latency.

A

D

Can see increased IOPS.

A

D

From the third to fourth wonders that occurs in the performance.

A

D

They can see increased decrease in the latency, so that's APO expansion, their.

A

D

To new twists were.

A

Also being produced yes,.

D

The performance we're also all be increased as a result. In the initiator side, we can use the a file path, plugging.

A

D

Can increase the Civic or you can see the increase in the performance, so the trend is a similar. The target site.

D

And we can see the latency it's about. The tester will now work a performance so for the backhander storage we use an upper octave ice. Actually it can was a big conductivity.

C

A

And for the tests that we have this standard.

D

Is that resistance is turn on target.

A

D

Initiator for the target site for the left side, suppose we all use the up arrow indicator and for the target and we use the kernels target so as PT.

A

Case target this.

D

A

D

Can see that blue on use SP TK target? You can see that the vacancy is there's a very.

D

Decreases I like about the 20 to 30%, so where is BTK target.

A

D

Initiator site we use the kernel, initiator or SPD condition. Hater. We can see the relative.

A

D

A

A

D

Can see the two solutions is the pure kernel solution target.

C

D

C

A

D

Can see the latency there's a very.

A

D

A

D

The shock decrease the pout thirty percent and.

A

D

An increase in improvement in the latency.

D

Sdk GDK: well, we use the different numbers for connections to revaluate to the performance that the workload we'll use is about a 4 KP randon right.

A

D

Sir 70% is rate. 30% is right. We can see that in the same hardware, environment with a sense of you and the control SPD key a target.

A

A

D

A

D

Two thousand five times that of the kernel, so it.

A

D

A

D

The same CPU configurations, the SBDC has a fewer CPU resources, but it can complete the same task so for HCI infrastructure.

A

D

Is very good for the deployment of the environment because you can save the CPU space for more running of the virtualization machine and for the BCP. Oh, so.

A

D

This saves a cost a lot, but on the condition of the improvement of.

B

The performance.

A

D

Is an ongoing process.

A

D

From using the third-party features.

A

D

Also want to use the hardware feature like the Intel.

A

D

Max generation 100 of GP, stupid Nick, you can support a DJ Kue.

D

On the caduceo, kohai I hope you, you can also still improve TR latency.

A

D

To 2nd solonius about the details about.

A

D

How to isolate it hard to be a cute.

A

D

Cute need the support.

A

D

Is some technical requirements like the kernel driver? We need to feel supported.

A

D

D

Equation this is also existing in some new kernel and to support.

A

A

D

Need to use the epoch lesson.

A

D

Need to deal with.

A

D

The same anymore, Nicci.

A

D

Whether the queue is hesita Nov Pio is either killed. We need to use the PS socket MVP and the to.

A

D

Input this option so for the hardware we also need sound support.

C

D

Support the filtering.

A

D

The ships shipment of the traffic.

A

D

Control the shipment and traffic and for them in ICS Orica. We are still tuning a second house new release of the ni c Nick I think this features were being integrated into SPD case software, so that when they.

A

Are using in news we.

D

D

A

D

A

Sauce and it's not there perfect so.

D

We need to cooperate with the linux kernel to have more interoperability with meaning to have this test. We need to continue to do this. The performance tuning attitude interpreter with a third party, yes to have a deep integration, and also we need to use the hardware features.

A

D

To provide better offloading API.

A

D

Integrate with SPG and smile.

A

D

Have a seed, SAPT k, MF solutions is.

A

A

D

Are focusing on introducing tcp transport, its realization and some details? I know he'll. Give us some performance test will prove that SBT cases.

A

Performance, it is good.

D

So if they use a speedy resolution, then.

A

D

Don't need to upgrade kernel and we can realize enjoy this benefit. But if you want to use the latest the kernel m8e tcp, then the stability version need to be upgraded to.

D

A last welcome to take part in SPD Creek community, so there's a different ways to participate to like propose some questions and a buck. And if you are very energetic- and if you want to ask work out a speedy, can you can offer some patch? Yes, that the end of my presentation?

D

D

On the we chat, we have the community to release some SDPD case, a technical information. Yes, we were releasing a new SPD key and tcp transport introduction and have more details in Edisto. Welcome to.

C

D

Welcome all of you to read these articles on the widget and give us some feedback. Thank you.

A

D

Three questions so.

A

D

A

D

P D K and Oh participe and the Colonel's realization.

A

But for you, the tcp, it.

D

Was the first two conducted in colonel? Yes,.

A

D

We use SPD colonel because.

D

For the analyzed of the package, so I think.

C

The SPD K, why.

D

It has advantages, it's because SPD.

A

K program is free work, so.

D

The framework is about synchronization, the parallel working mode, so under this programming condition.

A

D

With the incoming.

D

Debit, we didn't.

A

Secure to tell with it you.

D

Can you won't be switch between different ass of yours use, so there's a no.

A

D

Hey it's many many uses a new driver, but for users they also still need to use the driver of specification.

C

For the kernel further.

D

So the block is too long, but for SPD creamed asparagus users join drivers. It's unlocked. It's.

A

The unsynchronized.

D

Determines that even they use that kiss Colonel spins tcp, it.

A

Dude proposed better than.

D

Colonel, so, if we want.

A

To test a local and we need disk under.

D

The same physical configuration.

D

Spd case, the criminal performance is better it's 10 times than that of the kernel. So I think this is a very good pointer to prove that up the SPD case better than that we have tested in espenak his plans. We think that this video should be released, but our tester is the worse than your stone.

C

A hundred of wedding cake so.

D

This is to do with the environment. We.

A

Have tests tutor it's.

D

More to do is the pressure test. You can go to the I aware of this video game. There is dynamiting.

A

D

Am a baster report: our environment is the same as that one, so you can just refer to that environment. Yes, the parameters are same. You can just refer to that report.

A

D

I have another question: I can ask you in orbit.

A

D

Thank you. Everyone.