Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Running High-performance User-space Packet Processing Apps in Kubernetes - Abdul Halim & Peng Liu

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Running High-performance User-space Packet Processing Apps in Kubernetes - Abdul Halim, Intel & Peng Liu, Red Hat

With 5G on the horizon, networking is transforming around us. Network functions have already found their way from proprietary blackbox into servers running in Linux. The Linux networking stack simply cannot keep up with increasing demands for higher throughput and lower latency of these packet flows. The packet processing pipeline is now run in userspace instead, bypassing the kernel. DPDK provides an environment and API to run high-intensive packet processing in userspace. Many CNFs are developed using DPDK. A DPDK application requires specific resources from a host for guaranteed performance. Deploying and running such applications in K8s is always a challenging task. In this presentation & demo, users will learn about open source technologies and components and how to leverage them to deploy workloads that requires high performance networking infrastructure in a Kubernetes cluster.

https://sched.co/Uacf

A

Okay: let's kick this off. It is Thursday afternoon on the last day of the conference, but rest assured after you've eaten your lunch and you're sitting down in this room. We have a really interesting topic for you all, that's going to perk you up and make your afternoon wonderful a couple of things before I introduce the speakers here today in their topic. Please hold your questions to the end.

A

We will try and make time for them and I'll run around with the mic if we have time at the end, but please welcome to the stage I'm going to read this, because it's a long and complex title and I want to get it right for everybody presenting running high-performance user space packet, processing, apps and kubernetes. We have abdul halim from Intel and Poong Liu from Red Hat. Please welcome them to the stage.

B

Good afternoon, thank you thanks for joining this session, so this session is about how to run high-performance user space packet, processing, apps and communities, and what capabilities needed and how you do it. So this is this regular, legal and disclaimer, so you can read it at no time so the topics that we're going to cover is that we will first talk about. Why do we care for to running such application in cuban areas and challenges and overheads that involve, and how do you utilize the Nix, full Rundlett and we'll briefly introduce the DP DK?

B

If you are not familiar with, and then we will talk about the sample deployment that we use for for the demo, then we'll talk about all the capabilities that needed influences to run the DPD application and, as I've been a turing, is a big part of this demo. So we'll talk about that as well, how the soybean networking is done in communities then we'll talk about the soybean networking operator. That is how that simplified the whole process, then that will be followed by a recorded demo that we have recorded for in our lab.

B

So this is just briefly so something about Russ, so my name is abdul halim I work in an Intel as a cloud software engineer and I.

C

Everyone I'm Pandu from the city of office of right now, I'm living in Beijing and focusing on enabled and FA used cases with open source software. Thank you thanks.

B

So, let's talk about the motivation, so someone may ask: why do you want to do such things in Cuban areas? Why do you want to run packet, processing, application and communities so I think the 5g? Is there a great example of this kind of use cases? We are super excited about the 5g and there was some great talks and demos related to 5g already seen.

B

There's a nice demo shown by other side yesterday, like 5g demo in the keynote, so we thought that it would be great to see that what is required to run that 5g network infrastructure application in kubernetes and how we do it. So the there is some. These are the some of the fundamental promises that we expect that 5g will deliver by 2020.

B

So this is supposed to see a greater throughput that is like 50 plus Mbps for each network device and lower latency after high reliability and service for service, such as a health, and then we have much higher connectivity density and more user mobility as well. So all of these translate into the more network need to be more faster and faster. So as all you know, that network hardware already capable of delivering this higher throughput, but the problem always goes into the software stack that software stack. Sometimes it cannot keep up.

B

Processing that that throughput so for Fabio also depends on kubernetes. That's because for the speed and it's scaling, that is, it offers, and we already seen that that 5g is deploying on kubernetes. So we what we want to see is that what it is required to run this kind of application and we can eat less the full Nick villages and running this kind of application.

B

So this is this table shows the challenges that we see in this with the table at the left-hand side in the blue shows the nick abilities and right hand, side and the green itch that how much is time constraint put under the software stack to process this kind of throughput? So, as you can see that the smaller the packet sizes, the the smaller that time we get into process East packets, so to higher the packet rate, the lower timing to process and to achieve that Iiro packet loss network is stuck must be less than that.

B

The packet arrival rate, so this overhead in kernels, as you can see that for a smallest package frame, which is 85 bytes in linux- and there is a 67 nanosecond to process- and this is which is translated like over two hundred one CPU cycles and 3 Giga cpu, which is very small a time in comparison. If you want redo and a smaller packet and there's a it's like four can gig Nick. This is a really challenging for software is try to keep up with here.

B

You can see that what overheads and kernel introduced, so a lot of these cycles could be spent on context. Switching system calls interrupt, handling, lock and unlock to to copy the buffer from a kernel, respect to the user space and also there's the transfers. If there's a cache misses, then your penalty also much higher. So how do we utilize? The full rendered so there's the two options first of all, is by a try to improve the kernel stack and other is the bypassing the kernel.

B

So that's the bypassing the kernel is there's a different way of doing it. One is the DP D K, which is the data plane development kit, which is a library and Driver user space driver that offered to do this kind of packet processing in the user space instead of the kernel stack and there's also our GME and XDP as well. So in this session we focus on the DP D K, because DP d k is the well adopted library for implementing the BNF application.

B

So this is about the DP DQ DP is the open source project that offers a framework to to do the the packet processing application, also, as well as the polymer driver that runs in the user space as well. So I will not go much detail into DP here. There's a lot of documentation already covered that, so our focus is not there, but what we want to see is how deep it ek does that so deeply care basically do.

B

Is it takes the dedicated processor instead of leaving into the Linux reg Euler to randomly schedule in the different processor it takes? The data kills the CPUs assign it to the certain device and it the driver run in the poll mode. So basically, the driver constantly pulling the packet from the interface and giving it to the application directly and also it also offers the lock Lex synchronization.

B

So you actually don't need to do a lock and unlock in and also it is less the ring buffer as well, and the packets also present the batch instead of passing it single packet. So that also add some additional improvement but deepening. It also comes with caveats, so it's for guaranteed performance.

B

It requires optimal resource allocation and tuning, so it requires exclusive CPU to be given for this application, and also you need to give a direct IO devices for a neck, for example, and defeating also need a huge face memory, so the to reduce that he'll be means with a cache misses, and all of this also require new, my alignment.

B

So these are the key requirements and kubernetes that needed for to run the PDK application, and the challenge here is to resource allocation and management all of these, so the device and CPUs and the new melanin to serve so this session is about all this coverage that needn't kubernetes to orchestrate the resource, critical de Fuca apps, and how to do it so on this demo that what we throw in here is we basically, we use 2d PDK application deployed in the Kuban energies before one of them is generating the packets, the packet DP DQ package- in other words the DPD cat, has PMD.

B

So what we are doing is using a using a two interface using one of them is the Cuban interest control plane. They eat zero and other net zero. You can see that that is the data plane interface. That's basically going getting a interface, a virtual function from there, so I've been ik and test p.m. do also similar setup. It's getting another BF from the other port and basically package in general.

B

The generator generates packet and test PMD received that packet and basically just do a loop back and transmit back to the same port and received by the parent that package generator. So we want to see that we can achieve the full run rate from these applications.

B

So this is the platform that we utilized in the lab. So this is the old pro server with a two chakra system and we use the intel xeon cascade, like the latest, the process of their and also we use their access.

B

7Xs be 7, 1 0 into 4. Building. For this this is 225 2 by 225, big port.

B

So this is the example of that test. Pmd that the parameter is accept, as you can see that it requires exclusive cpus and it also need the I/o device, the danique pc address and also it requires the new my alignment as well, so you, you got maximum performance to do this kind of high throughput packet processing. So all of this, as you can see, these are the these- are the resource that you need to be given when you deploy this part into the Kuban energies.

B

So any orchestration platform need to ensure that these resources are properly managed and accounted for for maximum maximum resource utilization. So the record recovery is that needed in kubernetes is the exclusive cpus of key one. It should be able to give the workload exclusive, cpu axis and also huge page memories and direct device assignment before SRB kind of devices, so that port can get AG exclusive access to those devices well, and also all of this resource need to be aligned with Numa.

B

So you don't end up with in multi circuit system, that CPU and memory and one new market and and you device in different, no Mazzone. So there is a no crossover so other components that we needed. Is we this.

A

B

Or IV network that we implemented, so we need to have a SV networking curve village as well, for that we have a sort of a network device plugin that does the resource allocation and sorry. This was discovery and registration to the kubernetes, the cubelet and then I saw B, Network, S or BC and I. That does the the port configuration, for example. If you want to configure a billion or the rate limit, or things like that, you can do that through a sorry BC na and we also have the soybean network operators.

B

So there's a lot of configuration need to be done for the boat as well, really wise plugin and, as we see an IP network operator, simplify this process and also the for the party spec. You need to give the certain resource request and the network attachment definition as well. That's been taken care of by parties the app that you field, which is basically get those parameters from there which devices are given for those workload. It gets those parameter and what we seen that has PMB it added those parameters.

B

We do not need to go in the pod and figure out what device on what CPUs are being allocated. So that's F net util the framework provides those information and there is there, also sorry build the network resource injector. That basically adds those required resource allocation for the party spec as well, so we'll see them into detail later on. So this is the software configuration used for this demo so before, for just this is just for the reference.

B

So here we'll see that how to get the exclusive CPU for the water so for 2 workload to get the exclusive CPU, it needs to be run in the granted QoS so for guaranteed, cubes and you're familiar with kubernetes CPU manager. You need to request both CPU and memory request, as well as the CP workers must be integer values. So this is so and this example.

B

You can see that we put the CPU values for tubes, which means that we're looking for two exclusive CPU for the workload and for numeral einman twit, which is we have to work on also for top regime and sure that is part of now. We wonder 16 as alpha features so to operate manager. What it does is basically ensure that your CPUs memories and devices are all aligned with the same Numa circuit so to enable the topology manager.

B

So this is the feature feature get a you need to enable using the topology manager true, and also, if there's a you need to specify what the policy that hopefully Mandir will run so there's a four policies, the non best-effort restricted and single Numa bestie first doesn't guarantee that you will get the know. My alignment, but there's restricted and single Numa will guarantee that you will get restrict the gnome alignment of all of these resources. So in this example, we use the single Numa node policies to request a huge page.

B

So we you need to request as a empty volume, so the huge pressure will be. First of all, you need to create the huge space on the platform and then cubelet will be a register this with the kubernetes api and in the pod request. You need to request as a empty volume, and you need to specify the limits and requests as well that how much you want, but one give it with the capabilities that you can only have one type of a huge page.

B

This, even though you can create two kind of to two megabyte huge prayers or one gig of a huge place, but queue cubelet only advertise one type. So that need to keep in mind.

B

So this is this is the how the soybean network device plugin is implemented. This is the graph probably looked a little bit complicated, but, as you can see, the warrant part is the main three components that needed to enable a soybean networking. So, as you can see in the pod, we have the DES or a BBF added, as an additional interface doesn't replace the default. It's zero Network that for the Cuban, his control plane.

B

So for that we need a matter plug-in that can able to add multiple interface more than one interface in a part, so we used Maltese and basically I saw every device plug in does the device discovery and the resistor to the cubelet and cubelet then transmitted back to the equivalencies api and when port request through the extended resource it gets elevated when the port is created and the Maltese retrieved which bf was allocated and then it passes on to the so ABC and I.

B

Look in the network see Rd and that's where we see and I does the port configuration for that beer for this part, and so this is the steps I already explained, so a therapy device plugin works on the config file, so that config file defines that how many resource endpoint you would like to create. So you can actually separate out your SRA BBF with the multiple resource and point. You basically saying that this is this: is the type of device I want into this resource, endpoint name, so you can identify them easily.

B

So in this example, we are basically targeting beer from the two different physical functions, which is first port and the other one is from second port.

B

So after the device is discovered, the device plugin will register them with the cubelet and cubelet, then registered them as part of the node information which will be reflected in the Kuban, which is API server. So this information can be used during the port scheduling.

B

So this is several steps is like that so you'll, once the device plug in register the devices to the cubelet, then we need to create a network attachment C Rd, which is just basically saying that how would would like to configure this bf and then in the party spec? We need to refer to this network attachment C our DS. By the way, this theory is used by the Maltese, so Maltese is a natural plumping.

B

Plumping working group is standard defined for this network, C or D, and also we need to space the resource requests for that. How many we have we need for this network. So, as you can see, there's a lot of fish steps to just to configure the IP networking, so we need something that can simplify this whole process. So with that we have, we have developed the IP network operator. I. Think tango will explain you and.

C

So let me give you a introduction on the SRV network operator, so the SRA network operator is a tool to hide the complexity of configuring. The are we components.

C

Kubernetes a provides, some automation to the example, the lifecycle management of the SRV software component, like the servicing I plug in the network plug in and network resource, injector, and so on. It also managed the configuration of the asari software components, for example, the it will generate the configuration file for the asari us plugin and also a config the hardware in your worker node, which means as are.

C

We is a hardware related feature, so you need to set up the hardware for to enable that so it, the Ansari Network, to ask the network network operator will help you to config your hardware, as are we Hardware on the node? It also discover all the re capable of interfaces or kernels and in the end it can also, although the different type of a kernel driver to our node, for example, if you are using the UDK application your party to loaded, therefore, we have a whole grabber.

C

Instead of the default kernel driver and in at the the last day, it also generated an network attachment definition for the Sree Network you create hit. So the asari network operator is built for over shift. However, it can also work in.

A

C

Where nate is so, if you are you interested, you can just try.

A

C

Your cluster well in this slide, you can see the workflow of how to config the asari functioning, your cluster and how to consume feature so for user. You should only care about the steps in red, so first we launch sorry network operator. Then the operator will provision. Yes, are we network config tool, so the config demon we discovery, all the asari capable hardware or cluster and reported back then the cluster admin can choose on which node they would like to enable the SRA capability and actually can take the virtual function for this hardware.

C

So they can create a SR. We network policy. For that, then the operator will config the asari device accordingly, and it also will generate the configuration for the SR. We reverse plug-in. So if, if you need to load the kernel driver, I will also do that. For you, however, it doesn't take care of the outside. So if you need to enable the, for example, the VFL rubber for your water function, you have to enable the ome iommu, your the next parameter parameter, which cannot be done by the network operator.

C

So the search tab is to create a as a real Network. All the work function. You just created your node and in the end you can just create a user or the first three steps. It can only be done by the cluster I mean and at the end user can create. Their part manifests with all the information.

C

The next is the a manifest example of a sorry network resource. So here you can see that is a daresay went. One relationship to the network, attachment definition will be consumed by motors, so you can see that we can specify the name space or you like to create the network, attachment definition, and you can specify what kind of a Pam you would like to use for this sorry Network and you should also give a resource name, which will be a line, should be aligned with resource name. We used.

C

Network as our we network us froggy, also aligned with the resource name, we can see no measure well. The second customer resource definition is a sorry network, no policy which, basically, is you specify? What which note you would like to convey your sry devices and in this note there may be several is- are very capable devices so for for which devices you would like to select a fake virtual function and you can configure the MTU for the virtual function.

C

So all those information will be consumed by either the SR we network computing, so they can Hardware for you and or can be consumed by the operator to generate the config file for the SRO in network.

C

C

So next, this Y is the asari network, node state customer custom resource, which should be only maintained and updated by the operator itself, so I say classic enemy. You should not do any modification to it. It will be, and by the operator itself, however, you can get the information from it. So, let's check out the status first, the status will be updated by the asari network config, which running on all the worker knows. So it contains all the hardware information you may want to know.

C

So what kind of devices you got on this host and find it I'm to you the current mq, the the number of water function which can be config for this is and after you created the worker function, you can also find information about the virtual function of this.

C

So in a spec side in the spy field of this customer resource, it is the the desire state of node which it generated according to the network, know that network node policy we just mentioned in previous slide.

C

So, let's to the diamond, we have a record time of orders.

C

So here is on my cluster: in this node name space. There is no pod yet and here's my cluster. I have only one pair metalworking worker node here with 36 1 Giga. He paid a country girl.

C

Then let's check the couplet configurations, so the topology manager is still a alpha feature in 1/16, so we have to turn it on and we also need to specify the policy of topology manager and the cpu manager. Here we use static for the CPU manager and restricted for the apology manager. Now, let's start the operator, so after this operator is started, it will prove will provision the config, daemon and network resource ejector.

C

So here you can see that we we can find all the asari capable devices in in this node. So in this time'll we we used these two devices.

C

Now we applied the policy and the create the network, the I--sorry network customer resource.

C

So after creating the SRV network node policy, it will generate the configuration file for the assembly device Blackie accordingly. So here you can see that we have two resourceful. Why is Nick one? The other one is two and the the network attachment definition has already has also been generated accordingly. So there's two were created after that we can say that the the SR we say, I'm, plugging and I- saw in a forty was bragging house all we already written to the worker node.

C

So now we can see that the hardware resource has been reported to the kubernetes api server, how we create check out the manifest of the pod. So the first part is the test. Pmd part we assign the neck equine the porter function, pantun equine and for the packing generator part we ready in attach it to the function.

C

Now the part is up and running as scheduled into the path, so in the rest right side of the screen, we go into the part of test PMD. So here you can see that we can retrieve the information of, for example, the CPU which burned to this pod and PC address of the whole function. Now, let's check that they are new, my line, you can see that all the CPU we use for this bar noumenon one unless check if the leak is also yeah.

C

You can see that Anika also in the Newman, so we're in the same you might human old the unless started the test, um the patient, so we so in the left side we go to the part of package generator. So, let's start I can turn reader. Now we get the MAC address of the SPMD pod, so we use that for the destiny destination. Mac address of the traffic like to stand.

C

C

That the package size to 512 and cleared the statistic of test VMD.

C

No less startled traffic, so here you can see that in the pack generator the package has been sent and received promising from the acai over to function, and that's also checked the statistic MD. So here you can see that there's no package draw or loss rx.

C

So basically, what we know today.

C

Also, here you can find the video record a YouTube and also all the manifests we use for this demo.

C

Well still, there's something we need to pay more attention, for example the isolated or currently, if you specify CPU, isolated CPU in boot, parameter of Linux, it cannot manage by 90s and not many by Kubrat, which could be a problem since for a local use case, we like to save you dedicatedly for packet, forwarding and also there's some auto factory solution for this problem. However, we would like to see any more upstream or that.

C

None of them are integrated.

A

C

Already have second, there's no topology aware scattering for these things, so at this moment we have CPU manager and the commodity manager in curve. Let's add that for the QV scheduler it cannot. They don't know the information of a pneumatic apology when they do way to the scheduling. So in worst case, a may result in the past schedule to a valid.

B

So yes, so a lot has been done in the community support this telco and the fiber use cases in kubernetes, and we can say that current is 5 purity. We have seen the demo yesterday as well and it is been deployed in kubernetes, but there are some areas that still need some attention and we need the water. The collaboration and involvement with that I think our tread. The the presentation is finished is that the references you can check- and this is there some additional resources that you can see. There's all this component to discuss today.

B

You can get some application note and feature brief as well and those so thank you very much and if you have any question I didn''t we have time.

A

Wonderful, thank you very much. Abdul and bone they'll be around to take some questions, but please move towards the back of the room, so we can clear for the next session. Thank you very much. Thank you. Thank.

B