Kubernetes IoT Edge Working Group, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes on Edge: Bringing Your Code to Constraine... Steven Wong, Dejan Bosanac & Kilton Hopkins,

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes on Edge: Bringing Your Code to Constrained Places - Steven Wong, VMware; Dejan Bosanac, Red Hat; Kilton Hopkins, Edgeworx

The main goal of IoT and edge solutions is to bring the workloads closer to sources of data, events and transactions. In many cases this means maintaining them in places with network and compute resource constraints. This session will cover technologies and best practices available to achieve this goal. We will start by covering some common challenges of creating IoT edge workloads. We’ll discuss pros and cons of using traditional containers versus upcoming technologies such as WebAssembly and WASI. Next, we’ll explore mechanisms of deploying built artifacts to the actual hardware, ranging from using different Kubernetes deployments, to simpler container runtimes and specialized Edge platforms. We’ll also focus on the evolving state of running WebAssembly workloads using Kubernetes and how to apply best practices to IoT and edge use cases. We’ll try to provide general guidance on how to choose the best approach for your project with a practical demo of one of the possible solutions.

A

Hi welcome to cubecon 2021 in los angeles. This session is about running workloads at edge in a new way, using webassembly with kubernetes you're about to get updates from uh I'm steve wong with vmware representing the kubernetes iot edge working group, I'm joined by dan bosinek of red hat and kilton hopkins of edgeworks, but they couldn't be here physically, but they graciously prepared recordings which will be patching in and they're going to be online for the q. A section.

A

The agenda we're going to do a brief intro to what web assembly is, why and where it might be useful. Then dion and kilton jumped hands on and tried out webassembly with kubernetes and they're, going to give a report on what the current experience and future potential is like. We'll wrap up with links on how you can join the working group of people trying to apply kubernetes at edge.

A

What is web assembly? Well, it's an open standard for portable programs that can be written in almost any language. They are extremely portable. A web assembly can be run unchanged on arm x86 and other cpus and they're. Also not locked into an os web assembly was originally designed to run inside of a web browser in a safe way, but people discovered that it was also useful as a portable and efficient way to run sandbox code in other places.

A

Far beyond the browser contacts, these r, the runtimes that uh support web assembly range down to support tiny os, less microcontrollers. You know the kinds of things that might cost eight dollars and run for two years on a on a coin cell battery. um All these aspects make them attractive for running on devices with constrained resources.

A

uh So how does webassembly compare to docker well solomon hikes uh who's, often credited as being the inventor of docker, has been quoted as saying that if web assembly existed in 2008 docker wouldn't have been needed now. Docker is much more mature, and perhaps at this point you know locked into some decisions and use cases, but people have found issues with attempting to run docker containers at edge on low resource devices.

A

Web assembly has a few attributes that position it behind docker, it's less mature, it doesn't have the supporting landscape, but it's also got these attributes that you can read on the slide that potentially position it ahead of docker for these edge use cases.

A

I'll tell you that I got in here at kubecon on saturday for the rejects conference and by my read the hallway track here has pretty much a third of the cool kids talking about web assembly. This. This just has a feel about it that something big could happen here. um It feels to me a lot like what docker was like circa 20 2013..

A

Now one of docker's advantages is the maturity in the supporting landscape, but what, if the existing tools, things like kubernetes could be adapted to do similar the same kinds of things they do with docker containers, but for web assemblies.

A

uh Could that be made to work well you're about to see some demos where a couple people went out there and checked it out and the quick answer is yes, but is it perfect? No, um you know so it's hard to predict what will happen exactly.

A

This could be optimized and docker turns out to be the perfect orchestrator for web assemblies, or we could have a replay of the orchestrator wars that came about in the early days of containers. I'm not going to pretend that I've got the answers to that.

A

Another observation is that you know almost 20 years ago we had technology like mapreduce and hadoop, bringing bringing compute to the data.

A

In that case, it was to avoid expensive transport of the data and edge has something similar going on it, they'd like to avoid the big network transport cost, but a lot of this is about latency and resilience, and web assembly looks like a good way to bring compute close to sources that are dealing with data, sensors and data generation and event generation.

A

um So I don't know why am I wearing this crazy suit? I'm wondering that myself, it's pretty uncomfortable, it's a plastic halloween costume, but I think the potential here is really big. You know the the original uh kubernetes was designed for giant data centers, where you would pool resource and uh use that pooling to gain efficiency and operation at a large scale.

A

In a way edge is similar but different. You know it isn't really pooling resources, it's the opposite of it, but you're managing containers as a at a large scale. If somebody can bring about this whole big picture of putting code in webassembly and managing to manage and govern it at scale. This has the potential to not just be like the existing cloud. This is as disruptive as a new type of cloud, and I think that that there's a chance that could happen so in the demos you're about to see. I'm just gonna advance this.

A

If you can read fast fine, but I put it in the deck, so you can look at it later, but we're going to cover a lot of things so right now I'll bring on dion of red hat.

B

Okay, it's demo time, and today we actually have two demos.

B

The first demo will show how we can use container technologies at the edge today and what are the some some of the specifics of that use case, we'll talk about different uh cpu architectures and how we can allow our workloads to to access the peripherals, because our edge use cases most likely would like to to access uh some sensors or or or some peripherals- that's the main use case. So, let's start with the with the with the actual example that we're going to use.

B

This is a simple python script that uses the the dht sensor attached to the gpio gpio.

B

Interface and that script basically reads the the temperature and humidity values from the script and formats a json payload, and send that that payload to our uh cloud, the draw cloud in this particular instance, as we will see, we will see soon. So what do we need to do now in order to to build this and run this on raspberry pi? As you can see here, I am currently running an os x system, uh which is an x88 platform in this.

B

In this instance, and I want to build an image that that will run on our on a raspberry pi. There are multiple ways that that we can.

B

We can do that like getting a proper, proper virtual uh uh machine that will that will do the build, but, but fortunately, for example, doc docker buildex tool can help us here and, as you can see in this example, uh we can provide a platform as a as a parameter to to the build and build the image of a appropriate, appropriate architecture that can can be run on on an arm platform right. So.

B

So now that we have this, we actually want to run this script and allow it to access a gpio right, and for that we can do. We need to do do things. We need to run our container in a privileged mode, and we want to enable specifically a device that will be.

B

B

Accessible from our container- and in this case this- this is the the dev gpio0.

B

So, as you can see now uh in a raspberry pi console, we have a container that's running this this container and if you take a look at the the logs, you can see that that we are successfully reading the sensor from the container and sending the data to the cloud.

B

We won't spend too much time on the cloud today because it's not the main topic, but you can say that there's a there's, a kubernetes, uh iot cloud platform called draw iot that can accept a different kind of payloads, and this in this example, we used.

B

uh We use this payload, so you can see that that our our data are being sent from the uh from our device. Moreover, uh there's uh there's an example of the quarkus application that we can use to build our backend applications that will connect to the cloud. In this case the force application uses the mqtt integration provided provided by the by the pro cloud. So you can find more information about all this uh in the links that will be provided uh in in the slides.

B

So, finally, so now we can see that we have our container, that that is running. uh It's accessing the peripherals it accessing the cloud so how we're going to schedule that there are multiple ways and we we covered some of them in the slides of this session. The simplest way could be to provide a simple system, control uh system control settings that will allow us to to run this uh to run this every time we uh that a device reboots right.

B

So so we provide a simple system, control uh entry that will run our container every time. uh The docker service starts every time the the raspberry pi reboots. This is like a super, minimal, minimal, uh possible scheduling so to say of the system, but something like io folk using a lightweight uh kubernetes like like kts or or uh some other uh system like ansible, is usually a better solution, and that's that's a topic for for some other day.

B

That concludes our containers at the edge demo. For today and for a change of pace, we will create a second demo now in which we will uh compare the state of west sun this web assembly today, how they compare to containers, how we can run them uh in the kubernetes environment and if and how we can.

B

We can uh use them uh at the edge edge today, so in order to run uh web assembly payloads in kubernetes, uh we will use the classlet project, uh which is a are currently a cncf uh sandbox project, and the class basically allows us to to to add a node to to the kubernetes cluster, which will be able to run webassembly payloads in a in a vasey environment, as we explained earlier in earlier in the slides right.

B

So in order to start the demo, I will use my uh kubernetes on the desktop on the on the o6 desktop and, as you can see here, I have one additional node here, which is currently not not active right. So, in order to to start this node, I need to start crossfit by executing trusted command and providing some basic bootstrap booster configuration once the classic vez is running.

B

We will see that our node is now is ready and, as you can see, this is a special container runtime, uh which means that this this node will be able to run to run a vasem vessel payloads.

B

So with this we have our basic infrastructure. We have a kubernetes running with a with a board control, plane and and uh in the worker node that can run payloads, and now, let's take a look at the concrete payload that we want to run. So I prepared a small cloud events as the example.

B

This is. This is the project that will emit cloud events to to the cloud and and receive in parse responses, as as as a cloud events as well.

B

One of the reasons uh why we we are not being able to replicate the original project is that, as we said, the versi at the moment is is not, is not uh uh uh able to to access uh sockets or or peripherals like like, like a gpio, but I expect this to change in the future, and we will definitely follow up uh with a similar example that that will do this.

B

For now, we will use uh the experimental, http library uh and uh connected to the to the cloud events sdk, also written in in rust and and provide a way to to basically serialize the serialized cloud events from the from the http requests uh and and responses.

B

As you can see here, uh we have a uh our event builder that that will create create a appropriate we want to send and we can easily convert that into the http request that that can be used by the experimental library.

B

We'll then uh post that to the uh to the cloud and uh appropriately uh try to serialize uh and and uh parse the cloud event from the from the uh from the from the response right.

B

So what we need to do now uh to actually uh build this uh payload uh as a and run it run it as a kubernetes board. That's that's! That's the real question right. So, in order to do that, uh we we have to uh compile it uh with a with that with a special rust target. Hope.

B

uh Luckily, uh rust comes with a good support for for the multiple architectures, and uh what you need to do here is actually only install peasant, 32, vesic target and use that target to to build our to build our binary right. So let's try this. I mean uh I have the target already installed at my machine as you can see here. So that's all good. Now we can go and try to build, build our program, our our workload.

B

uh This will take a little bit, but as a result, we will uh our our cargo builder will actually provide a a buzzing binary that uh that can be run in a you know. You know in a vasey environment and one example of that environment is called vasam, wasn't time and that's actually used by the crossfit as well uh to to execute these binaries.

B

If you take a look at the payload now, uh one interesting thing to see here is is the size of this payload and it's only three megabytes, which will be important for our later later discussion.

B

But at the moment we, our job is not done yet, because what you need to do is actually to uh we need to convert this uh vessel binary into a container into an oci container that actually can be pulled pulled by the by the grassland.

B

Luckily, for that, there's a resin osci tool that can help us to to provide uh to convert this to appropriate image, as you can see here and and push that to to the container registry, uh it's important to say here that not all container registry registers they support all this, but the github one that I I was using uh is supporting it another. Another important thing is: you can see here that the size of this container is basically the size of the of the vessel binary.

B

So it's a three megabytes right and if you compare that to to any kind of container, even even though the simple one we we run previously, it's a it's. It's a it's, a quite a big difference.

B

There are tools that can help us to to minimize the size of all the containers, but, but even with all that, I I expect the size difference to be to be at least an order of my magnitude uh different, and that could be one of the big advantages of of using uh using basm in in agile environments, because it it really takes a a very little very little uh size of the container and it's easy easier to schedule over the over the over the networks. So now we have to schedule this.

B

This container is a pod, and I have an example of that here. So you can see we will pull our our image.

B

But the important part here is: is this the tolerations section and that basically tells us that this payload uh is, is a vessel to vasi architecture and that we need to schedule that or on a proper node that supports that uh uh that architecture and and what basically, uh this means is that when we applied this uh uh this pod and and uh add it to our kubernetes to execute it, it will actually schedule schedule the pod uh on our on our uh crosslit uh uh crossfit node.

B

As as you can see, as you can see here, that's that that's the the exact node that we that we started uh started earlier now we can see that if the bot is actually running properly.

B

And as we can see here, uh it is right, so we create our every couple of seconds. We create our cloud event and then we serialize it to a proper, a proper http request by the cloud event. Specification pause that uh to the appropriate the service and then parse the parse response back right and that that works properly.

B

And that concludes our vessel demo. Just to repeat, uh we saw how to use.

A

The interested time so we have time for q a and jump to kilton's, but he was showing a slide with resources and I'll close. The presentation with that. So next up, kilton hopkins uh cto of edgeworks and he's also lead of the ioflog project under the eclipse foundation and.

C

It's an industrial computer that is uh sitting inside your smart factory in that yeah.

A

I think this started playing mid.

C

Streams, let me see if I could it's an industrial computer that is uh sitting inside your smart factory in that situation, on top of the host debian os choose a webassembly, runtime and you're good. That's not the only thing you need. Something has to coordinate, play the role similar to what cubelet plays. So what is the agent that's going to start up, shut down and monitor the status of the webassembly modules? That agent is going to do your workload management and your workload administration at the edge?

C

So a lot of opportunities for that and I'm going to get into a couple more slides talking about some specific uh open source projects that are going in. That direction are going to add that capability, but you also need to have edge connectivity handling.

C

So here's a situation where a pure kubernetes environment might not be suitable, and this has to do with the edge not doesn't have to do, particularly with the web assembly, but just something to note in a pure kubernetes environment. You have what I would call the standard issue cubelet and uh the communication on back to the remainder of the system, the master and so on is going to be rather fast and frequent uh in an edge environment. That's probably not suitable.

C

What you would rather have is a lighter amount of traffic with a little bit more tolerance for breaks and connectivity or limited bandwidth, etc.

C

In some cases, pure disconnection I'm offline for two days, but I'm still listed as an active node and nothing no pod is getting evicted from that node because it's understood that it will be checking in uh maybe every couple of days. It's a really good example of edge connectivity, environment problem so choose an edge native technology. If you have an edge environment that is not suitable in terms of bandwidth and always on connection for being able to deploy directly your kubernetes environment out to that edge location.

C

Of course, if you run all of kubernetes on the edge um then including something like microcades or k3s, then it's different, because now you talk about land traffic, but that still may not be suitable for devices that are on low power. Wireless etc, etc. Just something to think about, and so then the last requirement is you're going to need a repository of the binary code, the wasm files similar to docker hub for docker containers right. So where are you storing? How are you serving?

C

How are you verifying the integrity of uh when you're going to get out into production and that scale? These are all pieces needed. So since we have the requirements now, let's actually talk about the types of edge deployments where where and you might want to use webassembly for the purposes of this presentation, there are two types of edge deployment that you need to worry about in the first type, it's no edge device, interfacing required.

C

It's all about latency benefits, bandwidth constraint, benefits security privacy- in other words, I'm going to take a workload, and I want to take it package it as a webassembly module, and I want to push it to an edge node that is close to where people will consume it. So on the lan as a a web interface for people to do real-time, voting interaction uh for some entertainment purpose- great, it's just a box on the edge. If it's on the network, it's fine, it doesn't have any specific interfacing with an edge device.

C

It's just just a network layer, but that's not that many edge use cases. That's just some.

C

The vast majority of edge use cases are the second type here which I've listed as processing data from edge devices here, you're, actually taking in some kind of sensor, data you're talking to a camera you're doing some actuation, maybe you're using the the gpio pins of a board in order to drive, led indicators or take in data from microphone. There's all kinds of possibilities here is where you actually need an interface layer, so wazi is going to be required or washi or similar. Some similar concept, and now the host environment becomes application. Specific.

C

What I mean by that is you can't just pick any old edge node. You have to pick the edge node that has the camera that you want, and so um that's something to keep in mind about edge deployments with webassembly or otherwise is that you have to have the right stuff in place in the right places on the globe in order to get the data that you want, and then you have to find a way to interface.

C

This is reminiscent of like with android, with uh the the hardware abstraction interface uh there, so the hidl, the hardware, abstraction layer, interface, definition language. How am I defining the way for um calls like open camera and git frame and so on to interface with the actual hardware, and so this is work to be done, because this is not ready. This is not out there universal ready for you to use if your edge deployment is this type of edge deployment.

C

Think about how you're going to access the special edge resources that you want to actually process the data from with your web assembly and then how are you going to accomplish that?

C

So what's the current state and the future? So what does wazi look like today? uh Super exciting? That's for sure, there's a there's, a link at the end of this presentation to look at a blog post about basically defining wazi. Here's.

C

Why, as you come help it's a great blog post it'll get you up to speed just one read, but currently it's about handling the core system calls things like files and networks, so like read, write a file, get access to the network and so on, and these things are going to take some time and the principles of portability and security are fantastic for building out rozzy, but it means that we're going to do it right and doing it right means you're not going to have a proliferation of a whole bunch of flashy features, which means access to like cameras and stuff you're not going to have that immediately because it's going to be done right and so do you need access to that stuff today, if you do, how can you find a way to interface?

C

Maybe it's best to wrap it as a container and then your webassembly running in container can have you know that stuff available there's ways to think about it, and um so, let's talk for a moment about edge specific challenges, so web assembly modules are the new microservices. What does that mean?

C

Well, microservice doesn't mean container, but it arrived those two technologies, so the architecture of microservices arrived around the time of container technology, well, any modular code that can be mixed and matched and paired together with microservices and in that architecture it's a multi-microservice world, meaning your application is likely to be built of multiple microservices.

C

So if you're, multiple web assembly modules now in your new architecture, how do you do the interchange data between them? That is a common problem in edge computing, because things are not always interconnected, especially between edge nodes that might be in different buildings. Different cities, different vehicles, something to think about, and how do you go from edge to cloud and so on with the data, not in terms of the administrative connectivity right?

C

What's my workload, but in terms of where do I send these bytes that I gathered from this camera and then now another edge challenge is dynamic access. I start a webassembly module on an edge node that needs access to the microphone. That's installed in the board.

C

Well, I didn't know I needed access to that until I discovered I have a microphone and that's when I deployed the workload and then the agent on that edge, node told me hey. These are the things we've got oh great microphone. We want that. We want to process that data in real time now, unless you've planned the entire host os to have all this stuff exposed through a washi or wazi light layer, chances are pretty good things will pop up and you'll need to allocate permissions, allocate permissions upon discovery.

C

This is a big challenge of edge computing in general, and you couple that, with the challenge of the not yet build out interface layer for webassembly and you've got a lot to think about, and that means an opportunity to contribute to all of the build out of this in the world, so jump in and help so last, but certainly not least, what is it going to be like to integrate webassembly with existing edge technologies?

C

The easiest path is to go where edge technologies already exist. Why would you deploy new web assembly edge technologies when building the edge computing world is hard enough? Let's, let's put it on top of or integrate it with. So it's really a choice of webassembly, runtime and that'll drive what features you have available because not all the runtimes are the same, and they don't also have a standard set of of features yet that eventually it will all be table stakes.

C

Every webassembly runtime will kind of have the basics, but now you kind of pick a runtime based on what it's giving you and what's been built out, seems like we're going to be operating, containers and web assembly modules side by side for quite some time, maybe even one wrapped in another, something to get used to it's probably going to allow us to fix a lot of things before we move completely to a web assembly world and the use cases of webassembly at the edge are going to drive these specific advancements.

C

So in other words, I want to access a camera on a battery powered smart camera device. I would like to use webassembly. I will do the work to build out what's needed for the value of my use case. Will I also build out access to the temperature sensor in the camera? No, I didn't need it, I'm skipping it right, so the use case determines where advancements are made and everything else will probably just have to wait until it can get caught up. That's about it.

C

uh So hopefully that was uh helpful and uh let's pass it back to steve wong to wrap up this uh this session thanks.

A

Hilton, so here are the resources that they use. There's a github repository with the code that dion was showing when they took this thing around the block for the test drive. They found these various open source projects highly recommended. uh The blog post that kilton talked about is the one stop to. If you're going to read one thing pick up on everything get up to speed: that's it um a few other things to read that are recommended, so this session was put on by the kubernetes iot edge working group.

A

If you're interested in applying kubernetes in ways that get jobs done at edge, that's a good place to come and discuss it.

A

Some people are looking for a solution that would run whole kubernetes clusters at edge, but some of the others are just using things like web assembly that will feed data and event streams into a kubernetes cluster up at a higher level.

A

You can download the deck and get the links to join. The group itself join the meetings that are held by zoom converse on the slack channel. This slack channel we're going to be using for the q a at the end of this session, so if you're watching this on the recording or we get shut down because we hit the time limit, go on that slack channel for this group and uh kilton and diana are on there right now and they'll hang around a little while after this closes, so you can ask questions there.

A

We do have a youtube channel with the recordings of the meetings.

A

These are speaker contacts we're on github. I think we all share those same github ids on twitter and actually the slack channel is probably the best way to get a hold of us after this event.

A

um So that's it. You can download the deck right here, that's the scad site for the conference and I have uploaded the deck. I maybe made one or two edits so later this afternoon. I might push an updated version there if you download it a couple hours from now, but this one is pretty close to what I just presented.

A

So that's it if anybody's got any questions and dion and kilton. If the questions are about their demos, they are online. Now I believe.

D

uh So I um I got a few confusion about the whole um context of running web assembly in a edge and the kubernetes situation.

D

So it sounds like the kubernetes itself becomes. A bottleneck needs to be resolved by things like k3s. First, then, you can run some more efficient container in web assembly. That's.

A

No, I I don't think that's uh that's what the situation is. What it is is kubernetes, of course, was designed to orchestrate docker containers and a lot of the assumptions that went into the standard cubelet that gets loaded on a worker node in every kubernetes cluster presumes that it's managing docker containers running on a docker runtime. What changes with webassembly is you have a different runtime, some kind of runtime that would run the web assembly and some of the presumptions that cubelet made are a little different.

A

So there is an open source project called crustlet that sort of convinces the this is an oversimplification. But, let's just say it convinces the kubernetes control plane that it's talking to a cubelet and it's close enough that you can utilize it to uh you know, disclose what workloads you want running.

A

You know under what kind of context and have these things scheduled to run on nodes running the crustlet instead of a cubelet, and then the crustlet uh knows how to deal with the webassembly runtime and gets that workload running. So um it should work with any standard kubernetes. I don't think you necessarily would need any particular one groomed for edge, but the kubernetes distros that are made for edge should certainly work for just fine.

D

Is there like production use case of this thing? You just mentioned the cross link, something.

A

Well, I don't know, I think personal opinion is that this stuff is so new that production use cases and a lot of those edge use cases are pretty mission critical. I wouldn't do it, but a lot depends on your tolerance for being a pioneer and exploring things that might still be a little buggy, an improvement.

A

Okay, we've reached our time limit. I'll hang around on the hallway a little bit and, like I say if your questions are about those demos, kilton and dion are online, and I would just log in and go on to that slack channel and talk to them through there. Thank you.