Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Panel: Introduction to the Container Orchestrated Device (COD) Working Group - Renaud Gaubert

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Panel: Introduction to the Container Orchestrated Device (COD) Working Group - Renaud Gaubert, NVIDIA; Mike Brown, IBM; Mrunal Patel & Urvashi Mohnani, Red Hat; & Alexander Kanevskiy, Intel

The Container Orchestrated Device (COD) Working Group, is a small group formed by passionate Container Runtime Maintainers and Device Vendors looking to solve many of the challenges Devices face in the cloud native space. Custom Devices are used in many industries (HPC, 5G, AI and Deep Learning, Data center, …), from single nodes accessing devices over the fabric to thousands of devices communicating over the network, the working group is helping enable new and exciting progress in the data center! Attendees will hear an overview of the issues the current ecosystem faces with device support, new and exciting changes happening in the different runtimes (e.g: the Container Device Interface) as well as in orchestrators (e.g: monitoring of Devices) and the roadmap ahead to enable the cloud native space to support exciting new workloads (e.g: 5G applications, Multi-node Deep Learning, …).

https://sched.co/ekDv

A

Hello, everyone and uh welcome to this panel uh introduction to the container orchestrated device group or cod working group.

A

uh My name is renee gaber, I'm a software engineer in nvidia and contributor to kubernetes, as well as the nvidia container toolkit.

B

Hi well, I'm mike brown, I'm I'm a member of ibm. I work on open source development in in a group that gets gets a chance to work all the latest greatest stuff. It's a lot of fun, I'm particularly a maintainer on container d oci projects.

B

I work with the signo team and in kubernetes and and.

C

Maybe a bunch of other.

B

Cncf uh projects.

D

Hello and I'm alexander kanyevsky, I work for intel. Ask cloud software architect. My main focus and areas where I'm mostly involved is enablement of different accelerator devices and specifically node resource management, like with complex problems.

E

Hi, I'm renault patel. I work for red hat. I've been working on uh containers for a long time been a maintainer of flip container, run c, o c. I run time spec cryo and I participate in signord upstream.

C

Hello: everyone, I'm urba shimoni, I'm an engineer at red hat working in the container runtime space and I'm a maintainer of cryo and participants ignored as well.

A

So, let's start this panel with a brief introduction to the cod or container orchestrated device work group.

A

We are a small group of device, vendors, runtime, maintainers and contributors as well as kubernetes sig members. This working group falls under the cncf runtime umbrella, uh sig runtime umbrella. Sorry- and this is because we interact with many projects um such as kubernetes, nomad container d cryo and other projects like cata or in the hpc space, singularity or saris.

A

The charter of this group really is to improve the support of devices in the color native space and across it.

A

What that really means is that we're trying to enable new workloads we're trying to smooth out the experience that you have as a user or as a cluster administrator, and we're also discovering and trying new ideas such as defining about the security boundaries for devices and what that really means. What does it really mean to try and enable new workloads? um That's a question that I'm going to be asking and looking very intensely at alexander from intel.

D

I think so now for such introduction of working group, and indeed the usage of devices in past five years or so is growing uh in the cloud and like variety of workloads, are utilizing devices. Nowadays, it's machine learning, it's data plane acceleration encryption compression, whatever things what can offload with cpus and devices which actually we are talking about just to name a few like we have nvidia gpus, one of the most commonly used in the cloud and from our from intel side.

D

We have fpga quick, assist, uh also gpus and some other devices when from different vendors we have smart, nigs and, and obviously all of those vendors who have a devices wants to enable them in the cloud and the way how it's enabled it's evolving.

D

If previously, it was just like docker run dev my device.

D

Nowadays, we users are expecting to have it in more complex setup, in example like kubernetes, distributed, orchestrated workloads and by the way encoders to renault and nvidia, who drove our current implementation of device plugins in the kubernetes, but actually simplified a lot usage of nvidia gpus in device learning. Oh sorry, deep learning uh model training the main thing actually, which it helped. It helped to change the mindset of a cloud users. What devices is not something rare, it's something it's a commodity.

D

What way can be utilizing inside way, workloads and changing that in the mind, actually brought a lot new subset of problems both from a use cases perspective and from user experience uh perspective. So if previously user wanted, I want a device. Nowadays, uh we are talking about a lot more complex scenarios. Like one scenario is we have a vmware based runtimes and we want to use devices where we have devices which have uh different properties.

D

So, for example, users say I want fpga, this particular bitstream loaded or we have connected devices where the smart mix says.

D

I want to be connected to specific network and fdma pipeline when we have a more complex scenario, when multiple devices within node is connected as a pipeline like fpga, nic, special isaacs and so on, and when the worst case scenario is something what in media trying now to to solve, is multi-node deep learning so, where you have a device, is connected between the nodes, and you have a problem with where device become a resource, not as a single node but a cluster resource and to utilize. What in the workloads we end up? What were existing.

D

All extension points is not enough. We need to do a lot more, uh the way how we describing devices for way how we preparing devices we have preparing a workload and how we run it and based on all of that, based on all the discussions, what we have in the working group, we realized that what were different apis between the cncf uh components needs to be improved. The overall user experience needs to be improved to enable all of this.

A

That's some pretty cool use cases, so special devices are now commodity. Are they special devices? Now? What do you think you're rishi I've I've heard about this core id called cdi, the container device interface? Could you talk to us about it? How does it helps of these different use? Cases that likes from intel just talked about.

C

Yes, that's a really good question, um so the container device, interface or cdi for short describes a mechanism for container runtimes to create containers which are able to interact with third-party devices.

C

So currently, there is no standard for device support in runtimes and orchestrator engines, for example, kubernetes and nomad both have a concept of device plugins, but they have very different frameworks for these device. Plugins docker has an entry plug-in mechanism, while fodman has the concept of hooks- and this holds true for other container runtimes and orchestrator engines as well.

C

This lack of standard results and vendors having to write and maintain multiple plugins for the different one time so with the cdi. The plan here is to have a standard way of supporting third-party devices so that the user experience is consistent, regardless of which runtime or orchestrator you use.

C

Portability between runtimes will be easier and there won't be a need to resort to various hacks for the different runtimes, which in turn means that maintainability will not be a nightmare or at least much less of a nightmare.

C

So in linux it is quite straightforward to expose a device node into a container for a simple device. All you need to do is pass a flag and the device path to your runtime.

C

However, complex devices like gpus and fpgas require much more involved operations, as alexander mentioned earlier, these operations can range from things as simple as compatibility checks. Can my container run on this device to device specific operations like reconfiguring an entire fpga or memory management and gpus?

C

Now the cdi is only concerned with providing containers with the ability to be aware of devices. Tasks like recess management is not part of the scope of the cdi.

C

This narrow scope greatly simplifies implementation of the cdi spec and provides good flexibility for runtimes and orchestrator engines. The cdi follows the model of the container networking interface. A json file is returned, written to a well-defined path on the machine. The json file contains vendor-defined information on device specifics as well as operations that the container runtime needs to support, and this information is used to transform the oci spec accordingly, which in turn, results in admins having a seamless experience while setting up dedicated devices.

C

So, basically, the container orchestrator device's working group is aiming to improve the supported devices in the cloud-native space, and cdi is our first effort in the field.

A

That's a really cool introduction to cdi. Thank you.

B

A

Much your rashi, um you just said that currently the space is really fragmented and that different orchestrators have different mechanism. Renault, you uh are a contributor to the oci, which is a standard spec. Could you tell us a bit more how you thought or how you think this is um doable, or this this is this use case is addressed today.

E

Sure so, if you think about the oci runtime spec, the way it works, is you write a config.json that describes how a container is created and run so think of the properties of a container such as its root file system like it could be based on ubuntu what you see inside a container all the files? That's the root file system, the process that you run.

E

It could be like a bash shell, the name spaces from linux that are used to define the container and the security aspects of it, such as sa linux capabilities and so on.

E

So when you, when the ocr runtime spec was created, networking was one aspect that was considered as out of scope and the reason being that there are tons of ways to set up networking for a container and different vendors would have different solutions at layer, 2 or layer 3 and the runtime spec didn't want to get in the way of trying to define every possible way to set up networking.

E

So the idea was to introduce hooks, which can then be used to join the network namespace of a container and then set up the networking as uh you see fit. So that was one use case and another use case I can think of is enabling system b.

E

So if a hook sees that someone is trying to start the systemd process in a container, it can set up the right mounts to write c groups for systemd, so systemd can start up seamlessly without any additional setup by the user, and the final most interesting use case uh is enabling gpus, so gpus have a lot of setup that are that is required beyond just uh adding the device node in the container like setting up additional mounts, running, ld, config and so on.

E

So initially this started uh with with a custom hook that performed all these steps like. But writing these hooks is not easy. You have to know the knowledge you have to know the internals of how containers run, how run c uh works with mount namespaces and so on, and then you have to join all these namespaces and perform these operations.

E

So it's a very imperative, low level bug prone approach, if I would say so- and there are also some security risks. So if you think about it, run c or any container, runtime is better suited to perform these operations. So writing all these hooks. You can see a pattern emerge that most of what these hooks are doing can be done by run c.

E

So that's why the idea of cdi is to be declarative, so cdi allows you to declare what additional changes you want to make to your container. Like you want to add a mount, you want to run a hook. You want to do some additional configuration. So, instead of your hook doing all these operations, your cdi is allowing you to make changes to the spec file, so run c can perform those operations for you. So that's how cdi helps and simplifies the status quo.

A

Thank you for this example and just explanation of what happens in a low level um talking about low level. I know that mike is a contributor to container d, and I've been following a bit that space and I heard about this new idea- that's called nri. Could you talk to me about nri? Is it the same thing as cdi? Is it something different?

A

What do you think.

B

I you know, I think it's I think it. It is somewhat different, but I think there's a huge opportunity right now to to integrate the the efforts uh between between you know cdi and the nri effort, as well as work with the signo team. With you know, with kubernetes at coming up with a you know, additional integration um we haven't mentioned much about the container runtime interface.

B

The cri was put together by by the kubernetes team to specify how the container runtimes should manage these these pods and containers, and and over time they needed more. They needed to manage the resources that are that are on these nodes themselves directly.

B

In some cases, they needed to monitor their health with probes and and they went around the container runtime and- and that became a problem in some areas, while we've all agreed to use cni for networking integration at the contender runtime level for the content, runtime integration, implementations um and we think that's been a good model, it hasn't been the way we've done all the other resources.

B

I think the the issue renault is that you know some teams see see this from a pod. Spec level or a container specification level and other teams see it from more from a low level. Hardware implementation level on the node, and how do I gain access to those devices I can use outside of a container inside the container right and renault certainly had a really good way to solve that problem with the hooks. um It is very low level um in in mr crosby.

B

You know who created the run c originally came up with this idea. After looking at all this and answered asking a lot of questions, it came up with an idea to do a new node resource interface. That would would provide for the capability to add plugins, that container runtimes would implement or be able to to be able to run that would actually access these hooks in such a way that the container runtime knows that they're happening, and hopefully with caps that will will do for sig note.

B

Kubernetes will also know what's going on and be able to manage those resources from a higher level perspective. You know by talking to the plugins through cri, a lot of work to do here um and a lot of opportunity for for the for the all the groups that hopefully are listening to this. We we need help. If you go to container d, uh slash nri, you'll, see you'll, see a brand new repository that michael created a bunch of code, some samples and we're looking for ideas.

B

It doesn't support pods yet, but that you know another pr will push that in so you can. You can actually create plug-ins to manage the resources for your pods and and containers using the hooks, underneath the covers that that renault and team put together in run c and hopefully will be implemented in the other run times again.

B

We'll have to work with the runtime teams, like like cattle containers, um to make sure that, when we at the shim layer, where we host these instances, you know for of vms or of containers that we that we've got. You know the ability to manage these hooks to manage these access to these resources in the containers, so that, when the container writer just wants to use it, it's just there and it's easy to use, get a better, a better user.

B

You know a better user feel when, when they create their pod specs and things just work so.

A

What I'm hearing from you is that there's a lot of work, but it's exciting all right. uh Let me go back to. Let me go back to the road map. I think it's really important that we talk about this group's roadmap to finish the presentation right so because we know intimately the problems we've decided to tackle it through a layered approach.

A

Right now we're solving this core problem, which is how do we expose a device to a container and the solution that we've come up, for that is cdi and, as brunel has mentioned, it's it's a very declarative approach uh and something that we are really excited to show as we continue through this process of solving these problems. um The next step here is the node level.

A

How do we select which device needs to be assigned to which container and the problem that we're trying to solve here really is when you have workloads that are very sensitive to performance, when you have workloads that have multiple devices talking together, you want to be very conscious about what cpu, what memory, what nick, what gpu, what if pga? What is it you're? Selecting if you have a cpu that is very far away from your nic or a nic that is very far away from your device?

A

You might destroy the performance to the point that it might not be even worth talking to that nick and so selecting which device needs to be assigned is a very difficult problem. The next step here is when you start looking at it from a cluster level, when you have workloads that need to run on multiple nodes and communicate together or when you have devices that are over the fabric.

A

Typically, you want to have these devices talk to each other that are talking to each other would be very close, uh for example, on the same rack, um and so the problems that we're trying to solve here.

A

They require um the ability to figure out where the right knobs to expose to the users were the right, plugins to create and plug in systems um and they're, really exciting problems, but also very challenging, and so um there's still a lot of space to be discovered here- and this is the conclusion of this panel- is that we are still a very new group and there are lots of really exciting and challenging ideas.

A

uh We'd, like people to contribute and to help us figure out where some of the solutions that could be done in the area we would like feedback. Some of these ideas are going to interact very very strongly with other ideas out there and um how do they intersect? um So, if you think that cdi might help, you we'd be more than happy to have feedback from you.

A

If you think that cdi might um be something that um that that you could be used that you could use, if you think that ci is something or some of these ideas that we're talking about you want to be able, you want to integrate.

A

Let us know we'd be very happy to hear about your use case and the way that you could, let us know, is go to the cod working group meetings. They happen every other week on sig, runtime zoom, and we're really waiting to hear from to hear from you um and with that said, I'm going to open the floor to questions from the audience.