Kubernetes IoT Edge Working Group, 10 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG IoT Edge 20210210

Description

February 10, 2021 meeting of the Kubernetes IoT Edge Working Group. Small group open discussion of jupyter integration with Kubernetes for an industrial control application. Brief discussion of trends comparing containerization to OpenStack/virtualization for edge use cases.

A

B

Can I hear you hello now it's better. I hope hello. You hear me yeah. Oh, can you hear me yep, hey, congratulations the edge today? Yes, that's terrific greetings. That's a great thing!.

C

So did you did you submit the paper.

A

Already, no, no, not yet, but we we are going to that's.

C

C

Will it be about ross.

A

Not exactly the ross is just a run time. So, okay, yeah, like a perspective for the event, is it's not exactly about runtime, so yeah yeah, yeah.

C

A

Use cases for the edge iot devices yeah.

C

That is great. That is great. I know I was very surprised when people from from cncf proposed to have that it's yeah, it's great, it's going in in a good direction, so yeah really looking forward to it.

A

Yeah yeah, how many do you think uh you are expecting like how many people, like uh hundreds.

C

I I don't know so so we are.

C

We are just tracking the the the papers at the moment and it's more, it's hey, hey there.

D

It does make sense to plug your microphone in, of course,.

C

It's totally for us right.

C

Yes, so I don't have any ideas about uh how many people will will actually attend the yesterday, but we will know more more information when we get closer by.

D

um I I just saw you saw your. She saw your slides yesterday. um Are you like building a docker-based, uh robotic system for for kubernetes? Basically,.

A

Yeah yeah, that that's that's something we already did I mean, like uh I mean uh the ros is, uh as I mentioned, with the end, the robotics is one of the runtime, so we don't specify the use cases with the kubernetes only for us, but for our aspect that includes ros. So, okay, not.

D

A

Exactly specified on the ross.

D

Yeah yeah, okay, because we're building something very similar but more specified for for manufacturing machines um where you basically uh split up your machine into multiple micro services and you kind of like combine all the drivers in in multiple container runtimes um and basically control your machine in a distributed environment. So you have like one one machine for your camera system: one machine for your other sensors, one machine for your laser beam, uh one computer. Basically, and then you deploy everything with kubernetes and you have like a job controller, which kind of just talks.

E

To all of these micro services and.

D

Controls everything that works like a charm.

D

And so I was, I was really interested yesterday when I just flicked through your your slides. Basically I was like oh right. It looks quite similar.

A

I'm happy to hear that.

C

C

That's cool, so what else has been going on with you guys?

C

We don't have any any formal agenda, so we can yeah. That's good.

B

C

A

Oh I'm doing all right, yeah there's uh I I actually. uh I am based in tokyo for now.

C

A

Yeah there's a lock down, but we're good yeah.

D

Yeah, we are same for us, it's like in germany, we're still in lockdown, but it's all right. So so you can still like leave the house and go to work and stuff like that. But uh of course shops are closed and yeah. That's pretty much it home office is getting boring.

D

C

Everybody is getting fed up with this right. Yeah.

D

Yeah, that's getting to the point where, where I think you are you can't just say it's just uh uh it's it's now, just just basically like like um that's lockdown is doing damage to your your industry. Basically, um but it's getting to a point where, where people are getting like a little bit crazy, you know like, like you, get like mental problems. Now you're like um it's getting weird so.

D

A

uh Actually, I got a question uh if I may, uh the answer.

B

A

Do you have like information uh or the you know, like a presentation slides in the past, uh related to the iot use cases? I am really interested in like the the activities in the past uh for the use cases for the agility with kubernetes.

C

Yeah we had something that at the beginning, but.

B

uh Let me try to find it nice, but forever no pressure when you have it uh yeah.

C

I'll send it to you yeah, because that was the one of the first thing that that we did as as a working group when we gathered this is try to define.

A

C

Because because, but that that was like two years ago, so a lot of things changed yeah in the meantime, so so yeah I mean if, if you, if you want to do a session about it, I think it's completely good to to have it all revisited.

A

C

Me try to find it.

C

And by the way, I also so here's the original uh white paper and it's still in a like an open edit mode. I was approached by by uh by by uh by a guy from uh from a cncf trying to you know or do a marketing around the edge day, and he asked if, if you are basically willing to to finish this up and get published, is you know a preparation for the event?

C

So that's something I I I wanted to to to raise that at this at this meeting. I hope still still would be here, but but we can discuss it, discuss it. Anyways there's a lot of yeah.

C

It would yeah require a lot of work, but it might make sense to to get it in into a better shape by the event.

A

Okay, I'm going to look into that.

C

And I think we actually did did a session or two about this.

C

At the early cube cons, but I'll try to to you know I gather material around it and and I'll send you what I can find.

A

Yeah, okay, but uh I think uh that the videos are should should be available in youtube or something right. Yeah, yeah.

C

C

Yeah, I really hope to took together some some good sessions and and get this thing, maybe a a regular thing for the future and yeah yeah the edge day. It'd be good. It's a shame. It's it's not uh it's a virtual event this time, but after all this pandemic is over. Hopefully we should get a physical one that yeah yeah.

C

C

Cool so yeah, maybe if we don't have any topics.

C

We can get on with our days and and continue yeah communication on slack.

D

C

D

C

D

Yeah, I I just like, if you guys want to, I- can show you like, like a short video of the interaction with our system, which we just filmed yesterday, but that's pretty much it that's all I have.

D

I just need to like start, my other laptop, um because one of our students just integrated our whole ecosystem, basically with uh jupiter hub. uh So you could like write, yeah, basically jupiter notebooks, to to to control your your your edge devices, basically, um which is quite nice, especially from a scientific point of view.

D

uh I just need my credentials. Give me one sec.

D

I hope she uploaded it. I'm not slide two teasing, you guys and then not.

D

C

C

Just reading about jupiter notebook, I I haven't come across before sorry, I'm just reading about the jupiter notebook. I haven't come across it before.

D

Okay, yeah, it's like basically a a very nice convenient way to to like like write python code in or whatever you want. Basically, you can, I think, even write c plus plus and stuff like that, um and uh why is everything so slow at the moment?

D

um Democrats are against.

C

D

Yeah, I think so.

D

Jesus, it's like really.

D

Really slow right now.

D

Okay, this works. This doesn't.

D

Yeah, so what we're basically doing is um kind of, like we have like one folder with our definition files uh for our.

A

D

So that's basically uh grpc or open api definitions, uh so every every service in our in our environment basically has their own interface definition. So here for like one for camera systems, one for for video cameras, one for sensors and so on. um These definitions can be compiled into into different classes. So uh what we do is we kind of generate a container with all definitions in them out of that?

D

So so you kind of compile that to python classes, and you then dynamically loads jupiter hub in to that as well, and you kind of just push that image that you just built into a registry.

D

So now, when you log in you have you can like like see um all the different you can access all the different interfaces.

D

uh We have another dashboard, which kind of gives you an overview of what is running in your kubernetes cluster right now. So you just use annotations for that just to see like okay. This is like a like a sensor service.

D

This is like a hardware service or or video camera or whatnot, and you kind of just see like what is going on in your cluster and you get the ip address, and then you can just like copy and paste the the the ip address or the service name of that into your your python code, and you can just dynamically connect to that nice part about that is, since you are basically running the jupyter notebook in the same name space as all the services you're not trying to connect, you don't have to like manage the uh the.

A

D

That's the authentication, because when you can lock into the jupiter hub, basically you're authenticated that you actually are able and you're allowed to use your machine. So you don't have to worry that much about it security, while when you like, just open up the ports of your services that can get quite messy, but, like I mean we have connected basically like like machines that can cut through metal and stuff like that, and I really don't want to just open up the network ports on that.

D

You know like to everybody you're, just like oh yeah, I just hear pc connect and let's go um so that that went a little bit. uh It worked quite nicely, but I really oh there we go.

C

I think nobody can share the screen at the moment because I'm not logged in as a. But if you give me a minute, I can log out and log in as a moderator and give you give you permissions to.

D

uh Give me one second, I think she didn't upload the video okay, sorry, so so it doesn't matter.

D

Why didn't she invite me well uh yeah too bad sorry for that.

D

But yeah like the basic workflow, now is you you have like.

D

You can deploy your your your containers with a with a deployment.

D

The idea is that we might need an our own operator for that, because we want to basically select nodes in our clusters and right now we're doing this with deployments because it actually allows that.

D

But um it's far from perfect, because you can still do like two hardware controllers on the same machine and stuff like that and that just doesn't make any sense uh demon sets could work as well. But then you run into the problem that uh you can't specify to have just one demon set in your in your in your system or you. No, you can't wait. You can't uh generate one service per demon set.

D

That's like like really weird, uh I'm not sure why kubernetes does that, but uh you kind of can't say like- or you can say like every node that has these specifications. Please get this this driver and this pot and so on, uh but you can't say, like start their own specific service for that as well, so I can actually access exactly that. Note, um not sure why but uh yeah um yeah and from there um you kind of can deploy your your stuff now in like seconds.

D

So you just kind of say: hey, I have my keyboard. I need this cluster. I just connected this usb camera. Then you kind of just label it like this.

D

This node now has this usb camera and then all the pods get deployed container or drivers get installed, blah blah blah um and you're ready to go five seconds later, and then you can just type in the the service ip address, basically into your notebook, and you can start using your camera system for your data science workflow, basically which speeds up stuff. Quite a lot like normally you would, I don't know, dig into I don't know 1000 pages of manuals for your hardware.

D

Basically you'd probably get a new friend on the on the manufacturer's side, who's like explaining everything to you and who's, like your hidden last for two weeks until you understand the the whole system. um Now it's just like oh yeah, I just deployed that service and I'm ready to go.

C

So what's the end goal is to have a live image or or to get a snapshot, or it depends on the on the on the paper.

C

uh Just curious, pepper, so you're saying you integrate with the with the machine and you get an image from the machine right.

D

uh Yes or you get an image with all interfaces of all machines running in your cluster, okay.

C

Yeah and and then you get get the live data from.

D

Yes, exactly so.

C

You can just like, like.

D

Throw up a website- and you can say like I need a picture, create.

C

D

Okay, for, for example, yeah you can create a dashboard or you can say like. I need a picture from my camera system and it will actually take the picture for you and then you can do all your python magic. I don't know integrate ai methods or whatnot and start working with that and when you're done, you can like say oh cool.

D

I learned from that picture that I don't know my process is running bad, so I might need to tweak these parameters and then you can send these parameters back to your machine and it actually incorporates that.

D

So it's like, like you, can do like a closed loop control, basically inside of jupiter kind of like that, um of course, it's possible to do it other ways, but it's just like super user-friendly, like you have like this web ide and you can start doing analytics right at the spot.

D

And and you can access the data directly inside the the jupiter hub um which normally takes, I don't know, yeah, you kind of run your uh you run your experiments and um you kind of collect the data and you have to open that up and then you, I don't know, train your models and then you go back to your machine and now it's just like everything.

C

Like clockwork, it's a it's a work, but yeah great I mean if you could share that video yeah. Just let me let me.

D

Let me have a quick run. It.

C

Doesn't have to be, you know, yeah yeah,.

D

Yeah um maybe later.

C

Yeah yeah, but even it can be a a a good session.

D

Yeah really, probably yeah, because now we are basically at the point where we can actually show stuff, um because the last few months went quite busy uh just setting up the infrastructure. Basically, and that was like the really first cool use case. um We already tried it with a few different, um different manufacturing setups, um where you, where you, for example, we had like a like a little foil or a this is called file.

D

um It's like a sheet of metal like really thin. uh I think 50 microns or something like that. We have like, like enormous amount of little holes in there, so the laser kind of like shoots hole in that and you want to to create a filter with that.

D

um The problem is that sometimes the holes don't get drilled through, um so you have like, like mismatches or some kind of of unporosities or impurities in there. um So so you kind of have like hole, hole and you have like a hole which is like, like not completely drilled, basically, um and what we did is just like, like taking pictures of that, then you can like detect the holes that haven't been shot through.

D

Basically and just repeat them- and it just just increases the the overall productivity or the overall quality of your products enormously with, let's say: 50 likes of python code- something like that and normally somebody would implement that and I think in c, plus plus and probably write his own. I don't know binary image, search algorithm, which is probably a little bit more buggy than the one from the opencv and stuff like that, and it's just like convenient so um speeds up stuff, quite a lot, but I'm not not sure.

D

If that is a way you want to like control your machines in manufacturing, because you kind of like have all these different moving parts you have microservices, which are not as reliable as like a monolith architecture.

D

You have kubernetes running in the background, which of course, is like extremely complicated, so you need more educated people to actually run these kind of systems which kinda gives us, or we had a nice discussions on that.

D

I think two two or three days ago, um like like, is that actually a viable way for manufacturing systems to run like, uh or is this just like way too complicated?

D

You could because what what I think with with all these edge devices is going to happen, is um you have kubernetes basically everywhere, but not everybody is able to to is able to actually control that or to manage the complexity of that?

D

um The question is, if that isn't a deal breaker for kubernetes on the edge, because it's just too complicated. If you know what I mean.

C

Yeah but would running just does the containers be enough for your use case, yeah yeah.

D

Of course, but just just get general so so we kind of like said: okay cool like let's, let's say you have like every machine in your job shop or or in your your plant is connected to kubernetes cluster and you do everything managed in the in the middle. um You can probably do that with a lot less people than you are actually doing right now.

D

But of course, if you scale that up to somebody like, let's say volkswagen yeah with like a massive amount of plants, you will not find that many kubernetes engineers who are actually if you could actually do that and then think yeah. So I'm a little bit worried about the brain drain. Basically so, um but you just don't have enough people to actually work with that and if kubernetes will actually come to a point where so many people can run it and then that it's that you're this is.

A

Getting like a like.

D

An operating system, if you know what I mean like everybody, can use windows and oh, it's gonna, get this this operating system for edge. If you know what I mean, um what's your your take on.

C

That yeah, I think we have some way to go there right.

C

But but yeah, so what about these distributions that that are that are like kts, that that should or or you know, onenote clusters and things like that? Would that help in in these these kind of environments, where you basically have just you know you you what you want you you want, you want to application deployment model of kubernetes. You don't want the whole infrastructure of kubernetes running on your machine. I guess right or or or.

A

C

One central cluster and all the machines behaving like like separate nodes right. Yes, that would.

D

Be like perfect, you know, um of course, the security aspect of that is a little bit tricky.

A

Yeah, because if you hack that.

D

You kind of like, like take down your whole manufacturing system, um so yeah.

C

I mean I'm a researcher.

D

So I can like yeah.

C

C

That's a joke yeah, but yeah. I don't think it can be as easy as as having like a you know: single machine operating system because you, you know you have to have a distributed distributed system, so things will break right and that's.

D

Yeah, I'm not sure if like because especially the distributed part is like it's so hard because things can go wrong in the unexpected places and if you don't have people trained for that, they just don't understand. What's going on and it's getting and also the whole ecosystem is just too a little bit too broad.

D

I think because you have to like you, have to understand first of all like docker, then kubernetes and and that's not even like, like you, don't even write a programming language, then you have like stuff like promisius jrpc registries, maybe harbor uh it goes on and on and on, and it's just like.

C

Yeah, it's a lot to consume a bit for sure.

D

D

But yeah, like the the question now is like how do you get that, basically back down to the to the people who are actually standing in front of the machine? Who are working with that? uh Of course, you can do everything by a web interface and kind of just like emulate.

D

A single note machine which is just like having this web interface and maybe or maybe he's like an ipad and controls. The machine or or whatnot, which could be, could be quite easy to work with. um But.

D

Oh, that was actually like a really cool use case. Anyhow,.

D

Coming back to the question is like what people in the background, do you need to to actually run that or to manage that and how many are they, because uh you would also like write the whole software for everything yourself right now, um for for the infrastructure yeah for the infrastructure and for the drivers and for the machines so yeah.

D

But of course you gain like flexibility. Basically so yeah.

D

What was it like we just had like? I don't know. I think one hour coffee talk about that.

C

No yeah, I see what you mean: yeah and yeah; no, no easy solution. That's for.

D

D

Yeah I mean how how do they manage that in uh in the telco industry? Do they I've heard that, like 5g and stuff like that, should be run on on kubernetes right.

C

Yeah, I I think the plan, so I I I didn't you know it's not my field of expertise, so so I might be wrong on many levels but yeah. I I think the idea is is to write a lot of software.

C

You know so so. Basically, all the telco industry started with like virtualizing the networks and things like that. So they have this system like for the base stations and things like that.

C

That's all going to be again virtualized and can be run as a as a containers so and and then that the point you don't from that perspective, you you don't care about the infrastructure, you you, you assume that you will have some kind of cluster and- and you know, depending on on the use case, will it be a a cluster per you know, per site or something like that and how that workload will be delivered. There is completely separated from from the actual logic of of of these, these components for the 5g.

D

Okay and what are they managing the network slicing or everything? Basically,.

C

I think basically, everything yeah as far as you know, but yeah, so basically yeah. You know translating radios to.

C

Starting from that layer up right, yeah.

D

That sounds like really interesting. Let me I.

C

Can try to try to find it, and maybe you you can get some some ideas from from that and and you know how can can be.

C

C

C

Yeah cloud native network network functions. Let me see so here's one link, it's just the first.

C

No, it's not that's not it.

E

C

I'll try to try to find them.

C

C

C

It I saw some presentations on on the topic, but it was. It was long time ago, yeah, okay,.

D

E

No, I I I'll get you deep. That's! Okay! No.

D

Pressure yeah, I just read that article I think, like eight or nine months ago uh about the topic, I was surprised that it took the telco industry actually so long to actually pick that up, um because normally they are quite fast, especially with stuff kind of like I think so.

D

Yeah um for me, like humanities, always sound like quite a good match, especially for like these. This problem, like some some customer, calls you and want to have like this network slice from. I don't know germany to india.

D

I don't know, and then you kind of just create your rule out of that and then give it to your cluster and the cluster doesn't sing and just kind of like connects to all the routers and then everything is basically set up. uh That sounds actually doable like.

D

I have no idea on how taco industry works, but it sounds like okay that could work kind of like you know. um I wouldn't want to manage that by hand. Basically.

C

Yeah, that's for sure.

C

I I think the telcos were heavily invested in the openstack, so uh so I I guess a lot of of that infrastructure is still managed like that, and you know then there's a long process of probably migrate migrating that to so so. Openstack is a little bit older right, yeah.

D

C

It's it's a precursor, it's yeah, basically, system for managing virtual machines, yeah and then the payloads on on the virtual machines. So it's one one layer top on top yeah. So so not containers but virtual machine space, but vms yeah and the workloads are on them right.

D

Yeah and a new experience, would you say that this is more currently bigger or smaller.

D

What do you mean compared to kubernetes like what's a bigger platform right now,.

C

I think yeah kubernetes overtook it in the last five years: yeah yeah, that's for sure, yeah.

A

C

And you know yeah, I think openstack is now more like a you know, legacy thing and and yeah.

D

So get got founded in 2010, it is already legacy now.

A

Actually, it's actually old right, it's like for me, it's like 10 years ago, so yeah. Now now it's kubernetes for us yeah yeah, but I think it depends on the use case. I mean, like a handling virtual machine is more. You can have the more isolation for the system, so using container. That's that if that container goes crash or make the system panic and eventually your entire system is gone. So you know, but if you use the virtual machine you can have the isolation.

A

So I think it depends on the use case how how secured you want to be or application or something yeah.

D

I'm just thinking about like like where this could be.

D

Like probably data centers right, like you, don't want people to crash other people's applications right, because if I control my whole system and it takes down two services- basically, it doesn't really matter because it just gets restarted right.

A

Yeah, I think it's once it comes to these iot devices. uh It doesn't matter, but uh you know.

B

A

Enterprise use cases, uh maybe enterprise databases or something uh sometimes you know we- we do need more isolation for that specific application. So in that case, maybe the answer is like a virtual machine, not container yeah yeah. I think it's always trading off so but the most likely the application service is uh running on the kubernetes these days, so yeah, as dan said, I think uh the kubernetes took over most of the use cases. I think.

D

A

Ever anybody worked with that.

D

A

Oh, what do you mean like.

B

A

With openstack.

D

Like like, how does it feel compared to.

A

I did I have done. I have not done the com compare uh between kubernetes openstack, but I was working on the openstack like uh that's a long time ago. Several.

B

80 years ago,.

A

C

A

C

Direct experience here, yeah.

A

As far as I see you know like for me, it's totally different thing, because the use case is different, so it doesn't get excluded for each other. So maybe some use cases you still use openstack or something.

D

Yeah yeah, because I can remember like when I started getting into all the cloud cloud infrastructure stuff. Basically, when was that I think three years ago, or something like that, um especially coming from an engineering mechanical engineering point of view.

D

I I was just like drowning in information um and uh I think containers container wars were still on, so you had like uh what was it mesos uh with with marathon and you had like kubernetes with stocker, and you had like openstack and, like I felt like every week, a new framework came out and I was like oh jesus and I.

E

Luckily picked the right one.

D

I'm sitting here today.

A

I think the main reason that container took over the infrastructure is like a using container, provides us a good uh friendly for you know like it's really so friendly to the application engineers yeah. So it's easy, so application engineer just cares about container and push it and everything is done and the rest will be taken. Care of equivalent is- and I mean like as simple as that, so the application engineer is the key to provide a service, so yeah yeah.

D

Yeah, that's that's what I what I thought at the beginning, like yeah.

C

You have like a really.

D

Nice, abstraction of of the house right um and that kind of appeals to people and normally the people that kind of like the developers make the technology decisions so yeah to take over.

A

That's exactly something I want to do on the edge iot devices yeah the application engineer friendly. So that's once the application engine right, the application. The container is done that should be run anywhere. It doesn't have to be the hdl cloud anywhere.

D

So you tell me that you don't like reading 1000 pager manuals before running your application or what.

A

It's a huge burden for the applications here, because every single time the platform changes or products changes they have to integrate their application. It's basically the same thing, but it takes time to integration.

A

So we we we don't like to do that. Nobody does. So if we support container on the agility devices, uh that can be done. It's done so uh you.

B

A

Application engineer just give me container and I will deploy the application yeah.

D

It's so funny that this actually happens, because um we like in the laser industry, like we, don't have a lot of people who can actually program really good um and in the german industry. There's like this one guy.

D

It's just like I'm not even sure if I should say that on camera, but um it's like this one guy who basically implemented all uh hardware controllers or like moving axes uh controlling lasers um for pretty much every uh vendors areas. So it doesn't matter if you, if you go to bosch or if you go to siemens or or whatnot, he has them implemented already so so. This is like this one.

A

D

It's like a like a company with two people and he's basically doing the framework for 90 of the german industrial laser industry. So so it's like this one guy who who writes the software for 90 of the laser manufacturing machines that are sold today so, like laser cutters, uh 3d printers uh ablation machines all the same dude. It's it's crazy! So, like I just imagine this guy getting hit by a train, we're basically.

D

That's like exactly the problem, this guy put the time in basically to to read all these manuals to like like get his get his this knowledge to like how all of that different stuff works. He knows like okay, when I want to connect to bosch axes, I need to do like.

D

He has written all of that code for himself and now he can can like save that, um but that's not a viable way you know like like. If you want to change something and this guy's, you can't get this guy on the telephone.

D

Thanks to you, you have to wait three months: hey because you're not gonna, implement all of what he did in like two weeks. I don't know.

C

Yeah, that's not a scalable module, no.

D

C

Hey tina, hey everyone, hey how.

B

C

B

C

We are basically approaching the close of the meeting.

C

I don't know is it showing on your calendar that it starts now.

B

Or yeah never mind yeah she's meeting great to hear you.

C

Okay, cool, I mean there's a lot of thoughts going on about about you, know, yeah, uh manufacturing and and and and and the in the cloud native and and the edge computing, so yeah. Something will come up out of there. There's a lot of interest. That's for.

C

C

Cool yeah so approaching the top of the hour so or it's I. I promise you to find you a little bit about the telco edge right yeah. I would promise you to send the video around the jupiter yeah yeah, that's cool! That's the takeaways, perfect.

D

E

C

Great yeah cool, then yeah see you around on on plaque and.

A

See ya talk to.

C

A

Yeah bye, guys bye, thank.

C