Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: There is a New Spec in Town: Getting to Know the Device Information... Billy McFall & Adrián Moreno

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

There is a New Spec in Town: Getting to Know the Device Information Spec - Billy McFall & Adrián Moreno, Red Hat

Cloud Native Network Functions (CNFs) will be the cornerstone of the 5G revolution. Unlike traditional workloads, CNFs require access to high speed network interfaces. Unfortunately, Kubernetes traditional networking does not fulfil this requirement out of the box. In order to close these gaps, a new specification has been developed called the Device Information Specification, which expands the existing multi-net specification developed by the Network Plumbing Working Group (NPWG). It not only enriches the information about the network interfaces, but also allows new accelerated technologies to be added and consumed by CNFs using an API-compliant library. This session will give an overview of the Device Information Specification and provide details of how it enables new network accelerating technologies to be integrated into Kubernetes through an end-to-end example.

A

Hey billy: are you ready for our meeting.

B

Hey adrian, I don't see anyone else here. Is it just yesterday.

A

Yes seems everyone else failed.

B

If that's the case, can we hold off for just a couple of minutes? I'm trying a new deployment and I need a couple more minutes to get to a breaking point.

A

Sure no problem, um I can help if you want.

B

Yeah I mean if you got time that would be great, um so I've got a customer. That's got some requirements, I'm not real familiar with they've got a legacy application. They want to containerize and require some additional interfaces to make it work properly.

B

Now I know, multicni will allow you to run multiple cnis for a container which will basically add multiple interfaces to the container, but the first time I've really had to do it with srv vfs.

B

I was able to bring up the pod, but I can't tell which of the differential interfaces is used for what.

A

B

A

Have a diagram or something can you show me what you're trying to do.

B

um Yeah, I think I got one here um all right, I'm gonna, I'm gonna share my screen hold on just a second. I found it all right sharing. Can you see the drawing? Yes?

B

Okay? So, as you can see from the picture, they need a high speed interface to receive some video stream in on one interface. The app does its transcoding magic and then sends a modified stream out a second interface. These video streams are at high data rates and need to be on separate interfaces.

A

Okay, um I have a 5g case. That is quite similar. um At least uh the interfaces uh interface requirements is similar enough. In fact, this kind of setups is more common than you think. um Let's take a look at your cluster. Do you have uh a terminal around? Let's see.

B

Yeah actually um I'll share, but let me send you some here: here's the login stuff, and um now you can maybe tmux in here. Let me share it. There we go.

A

B

Man, um so here's what I have so far. I found some srv documentation, so I've got enough that I was able to take my srv capable nic and divide it into multiple virtual functions so that multiple containers can share the same nic hang on. I got this command and somewhere one of my notepads.

B

All right there you go so now you can see um I've got these two interfaces with four vf's. Each.

A

B

So now, if I show you my queues, gl good pawns, all right, so there's my test pod and.

B

um If we go and look at it, so you can see here, I've got e0 for my default network and then a couple of net interfaces for the additional interfaces, but I don't know which net net one or niche two is used for what um so I'm logging I'm looking at logs and trying to hard code some values into my sample container, to make this work out. Okay, gotcha.

A

um I don't think you have to hardcode anything. Let me jump in real, quick, um okay and see what versions you're running.

A

So, okay, I see that you're running multis and the device plugin. So um let me see.

A

Let's start with the device plugin, for instance,.

A

Okay, you're running 3.3.1, which is fairly recent. That's good! Okay, I'm gonna! I'm gonna check the uh multis version. Just in case I wanna make sure you have all the changes um that we've recently added.

B

What changes are you looking for.

A

um Okay, I'm looking for some changes for the device info changes. The network plumbing working group recently applied recently approved the new spec called the device infospec. It specifies how additional device data can be passed into the pot. Look if everything is working. The information that you're looking for should be here.

A

Yes, there, it is, you see in the pots annotations, there is a network status field, and within that field uh there is a json object for each network interface. Okay, uh within that object uh there we have recently added the device info part, and that is the information of your hardware device that is nicely associated with your network interface.

A

Now as how was that association was actually made, it's cubelet, the one that allocates the device depending on the device, plugins reported pools.

B

um Okay, hang on that was a little faster, a lot of components. um Maybe we can draw a picture or something with all the components and how they talk to each other.

A

Sure, um actually, I'm working on some slides for kubecon. So let me paste a link in the chat. It's open.

B

A

B

A

Deck here, um let's just jump to slide seven. Okay, I'm.

B

Here I'll put it in the shared screen, so we can see it. Can you see it.

A

Yes, so um there in that diagram shows a typical sriov deployment. So, as you already know, multis is the one that allows you to call different cni plugins to attach additional networks right right right now. That part, I got yes, okay. Well, uh when it calls cesaro vcni, it adds an extra parameter with the pci address that it has to configure the pci address of the vf.

A

Okay, then uh the srvcni just configures the networking aspects of the vf, for example, mac address, vlan tag, uh the trusted mode and so on.

B

Okay, I kind of get that part. um I see the srv device plug-in in your drawing um now. What's the purpose of device plug-in again.

A

Yeah in the case of sriv, we need to make sure that we properly manage the limited hardware resources that we have in the node. In this case the virtual functions of our esri of unique. Also, we need to modify the pods runtime spec to provide access to those hardware resources, okay. Basically, that is what the what the device plugin does for us. um It is a demon set that sits on the node.

A

It discovers the available hardware, resources and arranges them into pools into resource pools, uh defined in a conflict map and makes these pools available to cubelet.

B

Okay, now um I do have a remember creating my config map um and I'm gonna switch back to the terminal here for a second all right. So so here is my config map that I created all right. Can you see that.

A

It looks like you set up the conflict map correctly, at least that config map is saying it's defining two resource pools. One of the resource pools will will pick up all the vf's from pf0 and the other uh resource pool will pick up all the vf from pf1. Is that what you wanted.

B

Yes, yes, so that sounds good. I think I got a little lucky there on what I copied so.

A

Okay, now, let's look at the network attachment definition: did you create one of those? Yes, yes, all right.

B

So yeah all right, so here's the yaml I used to create that yeah.

A

Okay, so, as you can see, your network attachment definition, references the pools that you've created in the config map, so that's ctx6, pf01, pf1, okay,.

B

Yes, I do. I do okay, so um here let me go back to that picture, so I can look at that again. All right.

A

Okay, here we go so so um having that reference um properly done, uh how it works is um when the network attachment their definition, references, the um the correct pool and a pot is allocated that uses one of those network attachment definitions, cubelet selects a free device from the pool and asks the device plugin to provide the device information.

A

The response contains information about linux devices that um have to be added to the runtime spec mount points and environmental variables, okay, so basically, that is how the pod has then access to a specific pci address.

A

Now the problem is uh that that information flows directly from cubelet to the pod and it gets to the pot in a very limited way. So you don't have like the context of the cni interface with it.

B

Okay- um and you said there was this new spec that was written. How does that tie in with what we're talking about here? Well,.

A

The spec changes this into the diagram in slide. 10 10 hold.

B

On all right, there's slide 10. Yes,.

A

There you go so, as you see now, the device plugin now writes a file in the notes file system that.

B

Is standard is that the device.

A

Info file exactly the device info file- okay, it writes the file in a predefined path, with a standardized format and uh once the file is written, multus will then pick it up and write it into the network status annotation under the right interface.

B

Okay, so this device file is actually on the node, so you're saying I could troubleshoot some of the device related issues just by looking at the follow. My note is that right, yes,.

A

That's correct.

B

All right hang on. Let me go back here and see if I can take a look at.

A

You know would.

B

You remember what the name of that file is or where it is.

A

It's in slash bar.

A

Cni, you know tap your way: okay, yeah um it device info, slash dp for device plug-in; okay, you have all the yes, oh.

B

Yeah there's a couple.

A

Of them there all right.

B

A

Format, so if you just if you come pipe it to jq, if you have jq installed, um so we can see the json okay content, correct.

B

All right that looks a lot like the device info stuff right there. That was in the annotations.

A

Yes, that's exactly right! So um if you look into the um annotations again, you should see that that json uh part is has been added to the network status annotation in the right interface. Okay,.

B

Yeah, let me go back here and see if I can figure this out: um okay yeah! So I see that pci address in the device info that we talked about and then there's that srov net zero that we're talking about in the name field.

A

Cool correct, so um the sro v net zero is the name of your network attachment and the net. One is the name of the interface um seen inside the pods uh network. Namespace.

B

All right cool all right! I want to stop sharing here. That's that was good, that's good!

B

All right! So um so we wait a minute. You wrote a new spec just so we could figure out which interface belongs, to which network.

A

No, that was just one of the added benefits. It allows network, um a new accelerated network devices to be added to kubernetes more easily. So recently I've been working on this technology called vdpa.

A

It exposes accelerated, virtio compatible interfaces, so it supports exposing a normal net dev to be used by standard containers, but it also supports exposing a special charge device called v-host or behost ddpa device that can be used by dpdk apps, for example, to do memory mapping basically to mem to map to map the memory directly uh from the nic to the application. Okay. So so, then they can dma directly into the app.

A

um So this is uh one of the use cases and, as you can imagine, without this spec, it will be impossible for the pot to know all the extra details needed to consume these new devices. Okay,.

B

So um so it sounds like it makes bringing up these newer network devices easier and also but more in a standardized way. Instead of using, I don't know, maybe proprietary environmental variables or something like that right right.

A

So we wanted to um have a clear spec to um basically make a pot on pots and application developers lives uh easier.

A

The spec supports a number of uh different network devices. These are iov devices like in your case mem. If ddpa and some more it's easily extendable, uh it's just a json format, so it's easily extendable and it should make it easy for new technologies to be integrated into kubernetes, even if they're not based on the sriv device. Plugin.

B

Yeah, but are there any cases where devices are created by like a cni plug-in instead of the you know, any of the number of device plugins.

A

Yes, for example, the host device uh cni and also the um user space cni that exposes virtual user devices.

A

The spec also contemplates that use case and it supports v-host user devices.

A

uh In that case, how it works is uh it works in a slightly different way, but um so how it works is the cni exposes a new capability, and if multis detects this capability, it will pass down basically the path where the cni has to put the device info file that we just saw. Okay,.

B

So you're saying that the cni instead of the device plugin now does all the work, but it has to deal with creating this file and making sure it exists, and then writing the file and take care of all that tedious stuff right.

A

um Well, yes, but most of the functionality has already been implemented in the library.

A

It's called the network attachment definition, client, and so the cni should just import that library uh to basically write the file and once that file is written, then multis will take care of the rest. Basically.

B

Cool okay. So all now all this information is ultimately available from with the pod right right. Well, good, all right! So now I just have to write a script or a small program to parse all this data in the container.

B

I'm not really looking forward to trying to parse json in a script, but I guess I can do that.

A

Well, I told you I've already been through these. um Let me send you a link you're gonna, like it. Okay,.

B

Open that link okay, I got it. Let me let me share it back over here, um all right, sharing all right, uh app.net, utility! Okay, let's see wait! This is a go module right. um I think the only issue is that everything I'm dealing with is going to be in c.

A

Drop down to the api section apis, okay, see there are go and c bindings. Oh yeah,.

B

There's go ansi. I like that cool all right, and um I see there are three apis um one to get the cpu data one for get retrieving huge page data and then the third one is to retrieve all the interface data. uh So has this repo been updated to support the new device info spec? We were talking about.

A

Yep, it's very useful. It hides all the complexities of parsing, the network status, annotation um environmental variables and system data, and it offers a nice high level. Api.

B

All right um there's a lot here. I see um test pods, uh dpdk, app container images, um yeah there's a lot here.

A

Yeah, it has some sample code on how to use um the apis. Oh.

B

All right, well that'd, be useful um all right. So what's this srv vf.

B

Development all right, there's a lot of stuff we've been talking about here. I see the network attachment definitions and config maps, um so um so, even if I decide not to use it, it shows all the different places in a container to collect the data and how to utilize it. This looks useful I'll start playing with this. um Do you have a list of all those repos that we talked about? Was it network, client or network something um yeah.

A

Yeah, I got a list, uh let me show you man, I I need to um invite you to my cubecon presentation, you're, just asking the right questions, so you have my um deck yep, so I think it's on slide 13..

A

B

Up there, okay, um I see yeah multis and um srv network device plug-in some of the stuff we've been talking about there. It is network attachment client, I see. Most of these projects are under the network, plumbing working group. um I thought that the network plumbing working group just wrote the multi-net spec that multis was coded to, and that was it.

A

Yes, that was some time ago, but they have recently approved a new open governance model. They have brought together many projects, including all of those and now it's a very healthy and diverse community.

B

Well, this is perfect I'll. Try to use some of this in my work, but you're going to be out. Aren't you if I can run any some issues.

A

Yeah for two weeks, actually, you still have that cucumber slide. Yeah.

B

A

Go to the next uh slide: okay and you'll see yeah there you go. Those are the upstream meetings uh you have links to the meeting docs and you have the uh meeting cadence there.

B

um All right I like it, but you know that's when they're saying I'm still like new to all this and learning all these technologies. I think if I went to one of these upstream communities, I just get lost and feel kind of out of place a little bit.

A

Everyone there was new to it at some point. If you have any issue just reach out to them, they are um a very, uh very kind and they will be happy to help you I'm sure.

B

Oh well good. That was real helpful. I really do appreciate it. I'm sorry for delaying our original meeting. What was the original meeting for I've already forgotten.

A

uh It's our virtual beer, clock zoom. Oh.

B

I'm sorry to delay that for work stuff, let me go grab a beer.

A