Kubernetes IoT Edge Working Group, 11 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG IoT Edge 20200311

Description

March 10 2020 meeting of the Kubernetes IoT Edge Working Group - discussion include KubeEdge 1.2 release and Knative for IoT

A

B

Yeah, so so two things like lined up from from our side, I I ran into a conference talk from one guy from redken talking about managing devices as crds, exactly what we talked about, and they had a workshop about that at one of the previous cube cons, and uh I invite whitey him to you know: do a small presentation of of of the demo and and the things they've done.

B

That, and maybe you know, spark spark a discussion about it because we talked about it a couple times, but this is the only time I see that somebody actually did a poc and and some working code in in in that direction, so it could be, could be a good discussion and, and the other thing is that we had a also a guy using hooking up eclipse honor, which is iot connectivity to do a k native uh layer and doing some things around that also on other topics.

B

We mentioned a couple of times so it'd be good to to you know present what I did so far in that area and and- and maybe you know, spark some interest and see what we can do in in in that respect as well.

C

Yeah, I'm definitely interested in maybe I'll bring some other people from vmware who are interested in those topics too yeah um the one with the device management through crds. If they already gave a workshop or presentation if they can put a link in the agenda, maybe I'll look at it in advance and have some more advanced questions. If I can take a look at it, a little bit ahead of time, I'll.

B

Put all these links in in the in the in in our general talks, I think, from both of the topics, and we can think, inter between us and and reach to them, to see you know, what's interesting and properly announced how to proceed with all or both of those, because yeah.

C

I think, and then the um the hana feeding into k native, I think what we were talking about there with k-native was specifically even a stream of events. Feeding um you know triggering k-native uh function. Calls so is that what is that? The angle on this that they'd be getting at.

B

I'm sorry, I was muted uh and searching for things yeah exactly exactly that, so so uh honor connected to k native and then a demo, basically having uh a video cameras taking samples as http requests, one frame frame at a time and feeding that into into a machine learning uh function.

B

Okay, behind the k native to do some recognition and things like that, it's it was yeah.

C

They can answer it, but uh I specifically be interested in how viable k-native is out of edge, because I've heard people talk about using the edge event stream, but then hosting function as a service kind of in a mezzanine chair that, rather than way out at the edge yeah. But I think it it all depends on what your application and resource availability is, which makes the most sense.

B

So this is all all about cloud site uh hooking up, so so it's not edge related. It's it's pure iot use case, so sending all the things to the cloud and and processing it there. So maybe something we can. We can see if we can do differently in in the future and expand this experiment would be to to bring more processing to the edge but yeah throughout the meeting I'll find links that there are a couple of blog posts on on the topic. So it can be interesting to the folks.

C

So I see someone on the joined with the initials qa, which I don't recognize.

C

I'm just asking because those edgenet people may show up during the call, so I wanted to see that might be one of them.

B

And yeah, maybe kevin wants to introduce himself as well. I'm not sure if.

D

Hi uh I'm kevin juan currently working on cuba age. I just joined to want to keep up to date.

B

D

B

D

I'm basing hanzo and currently I'm working in office, so I need to wear a mask.

C

C

C

So, aside from that, nobody came except for the edgenet which might not show we didn't have anything on the agenda. But if anybody here on this call wants to bring up a topic or just tell us about something interesting. You found lately and we'll see where it goes.

B

So one of the things I would like to discuss is how we proceed in the absence of the kubecon. So do we want to go and and maybe even intensify, our activities online?

B

Maybe you know continue with the survey and do something.

C

B

C

The goal of the survey was to cover the results at kubecon. I don't see any reason why we can't do that out of band. I think that the talk we were going to give there, at least in theory, it's still scheduled for as early as july. So I I just as soon leave that, rather than put it on the air and then have to do another one.

C

In terms of other activity for the group, um I'm not sure it you know bringing.

C

I think that we could do a better job of getting things on our agenda in advance. Now we've had a few that dropped off, but it almost seems like people who drop off are becoming common enough, that we need to have a buffer of maybe two topics for meeting to make this work.

C

Well, certainly, if you can bring that k-native talk, that's come up a couple of times at least, and I don't think that's actually been covered very much anywhere, so that that one is actually a really interesting topic and um I've got some people at vmware who are working on k native, not specifically for edge, but I think they'd be interested in that, and I can especially since the next meeting is in the north american time zone. I think that I can likely get them to show up for that meeting.

B

I'm just putting up the the links in in the in the agenda doc for the future reference, so yeah I'll I'll reach out to the to the to the folks who did it uh again, so they they were interested. It's just depending, I think, on the time yeah.

B

C

Think that's a big enough topic. We could go several meetings just because there's a once, you start feeding things into an event stream. There's a lot of potential iot and edge event, streams, yeah.

B

So I put it all all in there, so kubernetes of things talk. That's a a crd which a topic which I found very interesting and two blog blog posts about uh yeah hooking, up k native to to do an iot uh layer, iot layer, but in the cloud so far, but.

C

B

We can start from there.

C

And for a future meeting, I'd still like to get an update from cindy on um well, she had the project she was going to talk about, but even the azure edge sounds interesting to me.

C

I know that cube edge has its own cncf working group, but I haven't seen an update on cube edge myself for a while too so if people from huawei and futureway want to sign up to give us a briefing on what's new and or if you have any have had any recent releases or have one coming up, um maybe you have an agenda slot to cover that.

A

Sure we just had a release. 1.2 is a february.

A

So let me post the link. We have a.

A

And also we are going to build a uh last week last week, there's a cranial tsc meeting. We plan to build a blueprint project, basically show end-to-end things for the uh a cranial so to prevent the growbed into cranial energy. Let me post my release. Let me find that we have the release 1.2, there's a couple new features.

A

I don't want to go too much for the details, but we are moving on from uh now the kubernetes plan to have a three month release along with the kubernetes so about a month behind after the official kubernetes release. That's bill. Mia gave us about four weeks to update the dependency on the kubernetes, so we are just updated in the release. 1.2, we support kubernetes 1.117, so we plan to at the beginning of a main we are going to release a 1.3 that will support kubernetes 1.18 release.

A

Are we going to have a minor release between, and so the this major release in 1.2 is the agino auto residential registration and the kubernetes upgrade? Also we upgrade our gold support to 1.113, so we are going to use ula. We support golan. One point: 1.113.

A

The major upgrade is the cloud native transition reliability. We have a few uh support update. Let me see what else uh let me post this copying address in the chat, I'm putting my the change log in the chat window is the.

C

Okay, I see it yep, I'm gonna copy that over into the meeting notes too.

A

Yeah, so so, from now on, uh we are going to have a regular release for combat every three months. About I mean it may shave a couple days in case sustainability caused better stability issues and what else uh so yeah for for now on. uh We just had the uh commit community meeting today about three hours ago, so we discussed a few issues in one point: three release and the one of majorities uh one major things we are going to do is a cranial blueprint project.

A

C

Does the crano change much? I you correct me if I'm wrong, but I think cube edge started out more with a device, edge focus and then crano. As I understand it is more for software-defined radio management, or am I wrong about that.

A

uh I post another link is our uh blueprint. It's basically, we want to do a uh yeah. We are at the very beginning, it's more like lt edge, but we are trying to cover some of that enterprise edge or technical edge. So this blueprint is more like the offloading, so it's still working in progress. So the thing is we want to do some more marble, machining or inference offloading to the edge. So before that, there's a company called the mobile mobile edge is called mobile or ajax or mobile hijacks ox.

A

Your mobile ijx is a startup in san francisco. It's a dodge telecom founded. So what they do is they offload their machine learning directly to the cloud, but they have some restraints. So we have a new. We collaborate with them, going to use the kube edge to build a new offloading framework or blueprint.

A

So basically uh the mobile device will offload some inference to edge, as you cannot do it, it will offload the inference to the cloud and also the training will be happening cloud or the edge is all the offloading stuff. So we are going to build this reference framework in crano.

A

So hopefully that will help the industry to have a a sample framework.

A

C

Okay- and I I moved it over into the notes- yeah.

A

I was hoping so that's ongoing.

C

I'll take a look at it after the meeting, I'm just a little curious.

A

So yeah that's pretty much yeah. If do I miss something kevin kevin. Just.

D

uh I just want to add that uh cuba started with uh you, know, managing notes at edge and currently for the a bit longer term. We are we're trying to support managing clusters that age and what we're thinking about is that the uni uh cuba provided a unique thing.

D

Different to the other projects is that we we support the metadata synchronization between cloud and age, so for managing clusters at age. We probably will design something like federation, but diff, but the different thing is the federation, the federation kubernetes federation implementation uh kind of results in uh using list watch, which we think it's not a good format.

D

I mean not a good way over uh low uh low bandwidth, high latency uh internet, so you.

C

D

Sorry, you said it uses storage. I didn't. I missed the word.

C

D

The kubernetes federation could prefer today the the the connection between member cluster and its control plan is listed watch, so this watch is typically regarded uh work well inside data center network.

C

D

It can not work very well over internet, so we will probably uh re-implement that so currently, what we not yet decided is whether we reuse the api from cube, fed or not. That's one. One thing we are still uh investigating. Another thing is that we try to. We are trying to deal with the uh service service discovery and the service communication between applications located in different subnet or applications between cloud and age.

D

The hard thing is that you know that some sometimes the applications are in the private network, so the one first step will uh we will try to introduce the gateway for each uh typically, we have a gateway for each subnet and route or the requests through the gateway, so the gateway will have a public iep, so they can talk to each other through different subnets.

C

So let me see if I understand this so by cluster at edge. That's a full kubernetes cluster with it with its own control, plane, its own uh api server, its own ncd uh yeah and then potentially one of the issues with that is even version drift where you know who upgrades first? Can you operate of federations of clusters when some are at version 16, some at 17, for example, um for joint for interactive services when you say join the nets?

C

Are you looking at service mesh to do that, like istio or some other service mesh or just counting.

D

Yeah, so so currently we have uh two plan on this, so so uh because the user may not uh want to have service mesh or they may want to have a service mesh, but we think that if we, if they have service mesh, the functionality is much more powerful so for in the case with service mesh, uh we are trying to implement the uh the easter gateway api for that, uh if without service measure, I think uh kubernetes uh also are discussing about a gateway api. So we will probably follow that.

D

Otherwise we can first try to implement the ingress api age.

C

Yeah, the one thing I think in some edge applications that is perhaps you know unfortunate or problematic with using istio- is that um if you've got intermittent connectivity, these edge locations, maybe you don't want it to be http focused as your protocol, your pro, you know your prime protocol for that service mesh and if you could get something mesh like in terms of management and offloading writing the network and security code in your app, uh you know the sidecar thing, but maybe use protocols like mqtt or something a little more tolerant of internet intermittent connectivity.

C

I think that's that. That's something that at least in a lot of these edge things the world could really use. If somebody wants to build it out,.

A

D

A

And we just don't want to reinvent the view we want to leverage the existing. Let's see how it goes. Yeah.

C

There are a few non-sdo service meshes out there that have taken slightly different approaches like the one called. It is literally called network service mesh and a cncf project, and I think that was geared to go down to l2 and there are a lot of telcos.

C

I think who are looking at that may be participating in the project, um but it is, I don't think it's a message, cue like mesh or maybe I'm wrong about it- I'm not an authority on it, but I certainly am aware it exists and that in the telco space there are people looking at that one.

B

And yeah shameless plug from my side, the red hat, scupper project we presented here as well, which uses amqp as a as a messaging backbone for for doing this and basically going to in the other direction. Then the network service mesh on the level layer, 7 and and dealing with discovery of services on the on the on the pf.

C

Level it takes on all the features of an istio service mesh in terms of service discovery, security and observability, meaning it uh and load balancing. If there is such a thing relevant to it,.

B

Most of the stuff, because it it's done a little a a little bit different approach than than the traditional service mesh, but but it it does a similar thing in, in other terms, that that yeah, the so the service discovery, usually in the serious mesh landing, is done using dns so level three uh by by using something like scupper so layer, seven, basically, services on their own uh use, their own addresses to to subscribe. So it's it's a publisher, skype kind of kind of kind of communication pattern.

B

So so so you can lay over the different things. On top of that, but but you're not dependent on on. You know resolving things on on the dns layer to find things and- and you can do some interesting things- I I saw a couple of interesting demos like doing uh like multi multicast http, request response, where you know you're having those sidecar proxies to to do transformation between the the http request to a basically a message into the amqp network, which that is then multicast to multiple services.

B

Listen listening to that and you can do that once you're in mqp network right and then you get a multiple responses back and and you basically just uh do a multi-form response back back to the client, which is an interesting use.

C

Case well, I think if, if you have a sidecar architecture, that alone is a big win, because you know I'm a strong believer in the principle that your standard app developer, isn't it's tough to find an app developer, who's, also a security expert and a networking expert. And if you can split those apart, so that your app developer focuses on the unique aspects of the app there's, a lot less. That can go wrong and it tends to translate into a lot less security incidents and exposures.

C

So to me that you know, if you want to call it a mesh even for some specialized protocol. That's one of the key features that you want to have to you know to say: you're have istio-like advantages.

B

Yeah you're hitting your local http from your service right and you don't care. What's going on.

C

After that, it's interesting I read about, they recently had a new release of istio, and um at least the press coverage described it as going back to more monolithic like they broke it up into too many pieces and went back the other direction.

C

So, even though that istio is made to support microservices, maybe they went a bit too far on their own internal architecture, and I have yet to go deploy that to play around with it, but that that was rather interesting, and I still remember some people at kubecon, north america, making some snide remarks about how many crds sto required.

C

I think somebody called it a million which was obviously an exaggeration, but it was a lot and uh you know I guess that's another aspect- to keep in mind. If you try to bite off a mesh-like undertaking of handling networking for an integrated edge with many nodes, feeding back to cloud or mezzanine.

B

Did you looked at an announcement as well that they added uh web assembly uh except extensibility support in initial? I didn't get into the details, but.

C

Yeah, that's I. I really have just read about it and.

A

C

Gone, it's been a while, since I touched it, I I got pretty familiar with it last year because I gave a talk on istio mesh expansion at open source summit, japan. But since then I haven't gone back and refreshed what I have loaded in my own kubernetes.

C

B

Yeah it'd be good to try to find somebody.

C

Another topic I'm curious about it. I I really don't even know who we could get to talk about it, but um deploying kafka. Feeding. Ev you know, streams originating from edge is something that I'm kind of curious about. If we could get a user who's put that together or an open source project, doing that or.

E

We have done that. Actually, we build a little, let's say agent for manufacturing machines, which talks to to a kafka broker and the trick behind it is that you basically have this agent sitting on the machine and you can just dump all your data onto the agent you send it away and the the schema of the of the kafka message is in feared online, basically and then stored in different databases.

E

So you can just like dump video data on there. You can dump metadata, you can dump time series data whatever on there and then a flink process kicks in basically watches the stream and then channels it into the different databases and depending on the network traffic. It also applies different kinds of let's say, uh network compression techniques and batching. So when your average load and your network gets higher, um the kafka talks to the agent and says okay, we have a very high bandwidth situation right now.

E

Maybe you should start sending less messages and try to compress more and when the network is free, it starts. Let's say like freeing up all these resources and says: okay, you can go full latency. Now and just pump out everything you have.

C

So that's compressing! What for your application? It's compressing! uh It depends on the.

E

On the metadata or on the the on the schema, that is in field, so um it switches the compression compression technique, depending on what you put into the agent.

E

So video data might be compressed different than um time series data, for example, and you can basically plug in different compression algorithms in there, um which you can like like switch around. But this is like really early stage right now, so.

B

Yeah, but is there something running on on the edge that uh consumes data from kafka and doing all this forwarding or.

E

uh Well, basically, our edge device in this case is a manufacturing machine, so it's like, depending who you talk to it, edge moves.

E

So we are basically managing this agent with with kubernetes and right now we're still in the in the making of how to integrate it actually into the machine, because a lot of machines are running on windows and that doesn't play that well with uh with kubernetes at the moment. um So we are probably setting up vms on the machine itself to just get like a note inside the vm uh joining the cluster.

E

And then you have like this yeah, let's say on node communication going on on the machine and you can just just dump all your data around there and then it's gone and you don't have to worry about it anymore.

B

So but then kafka is not on the machine. Kafka is in the cloud right, yeah.

A

B

C

But in terms of feeding into kafka, I can appreciate this going back all the way to the 90s.

C

When I worked in industrial automation, I remember building into the product, I was working on an ability for data streams to do what you're describing as compression these were sensor readings, but it turns out that you've got a couple different problems when your network gets congested or when, even when, there's uh a bog down in the receiver on the other end, which can occasionally happen where, if you've got time series data, say that's at one second intervals, but you filled up any buffering, and you know you can't get it to the end point yet.

C

You'd prefer to take it to two second intervals, for example by averaging it down and then throwing out some of the samples and it got pretty complex where you'd even take a look at it, and if any samples were outliers which might might indicate alarm situations say they were several standard deviations out of norm.

C

You would leave those in the stream, uh but average down the rest of them so that you wouldn't lose the anomalies, and there was also a technique used of uh back pressure where the consumer could sort of like the old days of haze, modems with exxon x off protocol of telling somebody look, you need to slow down, I'm not able to consume at these rates. So please change, please, please downgrade your sampling rate and uh it it was actually a pretty interesting piece of engineering um there were.

C

The company ended up getting patents on it that, because it's so long ago, they're expired by now, but uh there was enough there that there was some real invention going on to get that to work for typical industrial automation applications, and I could see that how that would fit into kafka or some other stream of time series data.

E

Yeah, I think, like like the biggest problem we are having right now is just the data velocity. um We are basically up to uh three. We have sensors, which produce up to three megahertz per second.

C

E

And that's just so much data that you, you can't get rid of it um and that's like basically the biggest problem at the moment like how can we actually, first of all analyze all this data online um with camera systems which produce 15 000 pictures per second. So that's like enormous tasks to just write programs that can actually handle that kind of stuff, and then you want to get away with all this data and what I always see with kafka is like it's breaking down and around 15 000 messages per second.

E

From one note: it's just like crumbling down, I would say somewhere around. There depends on your on your on your batch size, of course, but um it's yeah. It's like a real challenge like what to do with all this stuff. Yeah.

C

And a lot of these protocols like take tcp are made just to engage in heroic efforts that, ultimately, you might not want with time series, because at some point you'd, rather all you always have finite buffering capability and at some point you have to come to the conclusion that look. I don't want you filling my buffers with all this stale information beyond a certain age.

C

It's no longer of interest anyway or I'd like to have it compressed down, but it almost amounts to a situation where you want to control the buffer so that, if the buffer built up rather than being in a protocol, you can't get your hands on. You want to go in there and and alter the data. That's been cute, making your own determination on what gets thrown out or replaced.

C

I think one of.

E

The key differences is like, basically, the the the push of data or the pull of data, like you kind of shift around the responsibility of who takes care of the data. I think kafka is like this. This broker system, so as soon as I put something on the stream, I'm not accountable anymore for this piece of data, because it's just gone, I I don't. I don't have to worry about it anymore,.

C

Other technologies, any of those that are pub sub too by dessert, they hid the consumer from you, so the ability of the consumer to give you this signal that look it's coming too fast. Please change sort of ruined by what's up protocol, because yeah.

E

um On the other hand, you have like these collector stuff like like primitive, prometheus or other scraping technologies. Like I don't know, I think fluently, for example, does something similar where you kind of just scrape the endpoint and there's a responsibility, of course lies on the on the device that is producing the data not to drop anything, so it kind of depends on where you want to put your buffer.

E

Like do I want to put my buffer onto the cloud with kafka, for example, or do I want to put my buffer on the on the edge device and handle that for myself, so um yeah like these two differences, of course, depending on the data stream, you have, you can have like hybrid ideas. You know.

C

Yeah but anyway for a future meeting the the I think we could do a meeting on kafka. I don't know if you'd want to do it or if you know, authorities out there, we could approach and recruit, but uh that.

A

C

Interesting topic and it would feed potentially into k native ultimately too, because I think a kafka feed to k-native is a pretty.

E

Common act yeah, we have done like uh kafka in combination with flink. I think um I'm not quite sure what they did with kafka internally. So we did like, like a comparison of a kafka compression mechanisms and.

E

Our own compression techniques, but I'm not so I didn't set up cuff card myself, so I'm not quite sure what we did there um so, depending on what you want out of this presentation, I might pass to somebody else.

E

Yeah, so I'm not that firm on kafka itself right.

E

B

C

Well, we've still got 15 minutes to go by the schedule, but does anybody have anything else they'd like to discuss or chat about put it out there? Otherwise, maybe we should just end it a little early.

C

B

That's all for me.

C

You've got the recording, so if you want to declare it until next time go for it.

B

Yeah yeah see you all in two weeks: okay, what else to say yeah see you take care everyone.

A