Kubernetes IoT Edge Working Group, 14 Sep 2018

Previous Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG IoT Edge 20180914

Description

September 14 2018 WG meeting presentation on KubeEdge by Cindy Xing followed by Q&A and discussion

A

B

Okay, so today I I'm I'd like to talk about uh edge project called the cougar edge. We worked on at huawei, uh so for this project we have two topics accepted by kubecon at shanghai this november.

B

So, as I mentioned, I like to share the ideas and the cup of poc use case and how they huawei view the edge cloud and the high level architectural design, especially for how I use kubernetes at edge, feel free to interrupt me anytime with questions or comments.

B

Can you move to next? You know like artificial intelligence. Nowadays is a hard topic and ai at the edge can enable a lot of possibilities such as municipal security and smart city.

B

I'm sure all of you have used the ticketing system at a parking lot. It requires a person to manually acquire and verify the ticket oftentimes at a busy time.

B

People are frustrated to find a parking lot or a vacant spot at a gas station, for example, the below picture. You can see a dashboard at costco gas station. So what I want to say the scenario is this image recognition or intelligent video camera installed at those public places?

B

They can enable a lot of things, as I mentioned, for the first scenario, the car and the driver can be recognized automatically and with actions taken and then for the second for the parking lot or the gas station the cars their locations. All the data can be collected analyzed so that a lot of things are guidance guidance. A simple guidance board can be shared to the audience, and then you enable a lot of efficient and then drill down all the tools scenarios we found out.

B

No, the below functionalities uh like we need to as a user, I need to register or configure the edge or the things at the cloud, and then a user need to configure, deploy the image, recognition, ai model or server list functions from a cloud to edge.

B

On the other hand, the functions would need to be scheduled triggered upon conditions on on the edge. Furthermore, all the image data can be collected and streamed from the ad to the cloud for advanced analysis.

B

We need to move to the next slide.

B

So here is a real customer scenario. We worked with a customer in china, it's about a water tank system in a city, okay, as you can see, there are abc three water tanks, geo located at different locations in the city and then for each water tank. There is the edge node, the sensors and valves are installed among other edges.

B

They are aware of each other, so they can from a network or perspective. They can communicate with each other, for example, water tank a can broadcast its water level to the other water tanks as a whole. The system can intelligently decide how much and the valves need to be open or closed so that the whole system is efficient and prevent like a water hammer, those kind of things.

B

So some details is like they need network communication among the water tanks. The whole system is decentralized in the sense in case there's no network connection between edge to the cloud the system can autonomously work effectively and local quick in case of, for example, I mentioned the water hammer scenario.

B

Actions can be taken really quick and reliably, so this is uh and on the other hand, from a cloud perspective uh adam for the whole water tank system. They can uh monitor uh the health for the whole system. So this is another scenario next, so I like to share how huawei is viewing the edge. As you can see, uh cloud is in the data center. Then we have the edge nodes and then the things a lot of iot devices we see edge is an extension of a cloud.

B

The edge nodes are located at edge, but they are managed and registered from the cloud applications or serverless functions are orchestrated and deployed from cloud to the edge. So this is as that, as we see it's a cloud extension and then between the edge and the cloud, there need to be a bi-directional network communication, the communication sometimes for the edge it's it has its own private network and then we need to make sure the bi-directional communication is enabled.

B

Secondly, the network connection is not reliable; sometimes the edge can be offline, and then we need to handle that the third will be. There will be large scale of edge, so we need to make sure the cloud has the sufficient bandwidth to handle all those.

B

Thirdly, we see the edge and cloud need to be loosely coupled, as you can see the water tank system there they want the edge nodes can be autonomously magic, manage them work if run themselves. Local, quick and reliable.

B

uh On the other hand, a decentralized system, the edge and edge can communicate with each other without even being managed by the cloud. Okay, so, lastly, but not list so the edge node itself can be heterogeneous.

B

It can be a regular server machine or resource constrained, raspberry pi, and then the edge itself would need to support again a large scale of iot things and between the thing and the edge uh there can be multiple communication protocols.

C

So I might be taking questions now or should I wait till the end.

B

C

Okay, so just a comment on the heading for the last one where it says heterogeneous, I think you mean non-heterogeneous, then right so you're, implying that the central could be say, x86 and then the edge locations, if their pies would be arm, processors or something.

B

Okay or you even have a gpu.

C

All right and then by different communication protocols, you mean, like non-tcp,.

B

Yes, like I'm qt or uh I think, uh like uh okay,.

C

I got it yeah edge to edge awareness. Does that imply that two two edge locations at different sites can talk to each other, potentially even when the central location is down and that they.

B

Discover themselves.

C

B

I got it thanks: okay, yeah, um what I mean is the the etched nodes. The machines can be different architecture, heterogeneous or even involve gpus, or they have different scale settings. But we as a goal we like to make sure our system could uh generically enough and cover all the scenarios. So.

C

How far down in terms of processor, do you envision these going? You know down to say 8-bit, cpus uh and really small things running non-linux operating systems, or are you at least constraining this to be linux everywhere?.

B

uh That's an excellent question, so I think from uh os perspective. Right now we are targeting linux, but uh I know like for amazon: aws is supporting our free rtos, but it's still linux based and for microsoft is exploring windows. I think this system, the you know like for kubernetes, it's supporting any architecture, it's just a kind of execution work.

B

The architecture would allow that it's just time or how they enable it, and then the gpu, all those things can be included as well. Rbd kubernetes supports that as well. Okay,.

C

I know you said it didn't necessarily have to be tcp, but is that uniform? Are you picturing, a mixture of say, classic internet plus cell networks.

B

Okay, so from an edge to cloud perspective, it still need to be internet, tcp or http, but between the edge and iot it can be different protocols. So then the gateways need to be, or the brokers need to be available to support albums.

A

Okay, thanks so in in this case, I see the picture on the right side. So are your architecture supporting both connecting things directly to the cloud you that simple connection and video or is it a transition from the one to to the other.

B

Yeah, okay, so uh for the open source uh project, the kube ad, we are sharing in kubecon. We are targeting the second one, but uh uh as a huawei production, uh we do plan to support the first one as well. Yes,.

A

So, together, working together.

B

Yeah yeah thanks, okay, so next and we can move right: okay, okay, so um here I'm just sharing a very simple version of the whole architecture design, and I also like to share a vision we see about this coop edge project is we want to see everything is just like you work on a kubernetes cluster.

B

Even the resources are located, the edge node is located remote, and then uh we have a special component. We call it uh cool bus or coupe edge bus. This one enables all the communications bi-directional and multiplex communication tcp communication between the edge node and the cloud and even like in production. We support like between edge to edge as well.

D

Is the edge controller there, a a kubernetes controller like it's a custom? Okay, it's a custom resource controller.

B

Yeah so I personally wrote the edge controller, so the the point is to address the loosely coupled scenario, because uh you know like nowadays when they use kubernetes covalent needs to every 10 or 30 seconds need to report a heartbeat to the uh coordinates api server and then, as I mentioned, that edge node can be offline as well.

B

So that's why we designed this edge controller so that it can store all the customer requirements like parts back to a storage, and then this storage can be synced down to the edge node based on the network connection and so that it can store it locally on the edge and then the edge can be autonomous autonomously.

B

D

And the um so this is a this is a fascinating slide. Hopefully we can maybe kind of talk talk through some of the details on it. How okay? How does that edge controller? Compare to the virtual cubelet idea? Does it does it represent? How does it represent the edge in the in the master control plane? Is it just a custom resource, or is it actually a proper node.

B

uh It's a proper node. It's still the same, so it's just uh hide all the from a concept perspective. It's similar to the virtual corporate, though all the at each at node is represent a single kubernetes node. We don't like hide anything there, so each individual node will be a single node here in kubernetes api server and.

D

So there's a so there's a second uh there's like another arrow to the control plane, which is the standard cubelet to master connection. In addition to the sync service or.

B

Yeah, okay, so uh I think I think also okay, so the app engine is a revision of kubelet. We wrote so the this app engine can take all the products back information and take and do things locally.

B

Another point is we want this app engine to be lightweight because as much as possible cover later has a lot of functionalities, we may not need it in the edge scenario, so we modified it and then it can take action. The other hand is as similar to equivalent it can do the plaque actions and report all the node and part status uh back through the sync service to the api kubernetes cluster.

C

Is this essentially like running static pods, uh where you know a static pod is one where the definition of the pod resides.

B

Okay, uh I I think it's different okay, because uh for static uh part, uh there's a shadow or mirror in the kubernetes api server. We don't do that. Customers can still dynamically uh put in their parts back and then we just regularly make sure they're synced down and.

A

B

Outline scenarios things will work locally.

C

So I'm also interested then in how the container images get pushed down. Are these held in an image repository, that's centrally managed and then pushed to the locations, and then what that implies, which is interesting is that since you're non-homogeneous, meaning the images that would run at central are likely x86, but the ones at the edge could be arm and making those in a an image repository is something that I'm not sure has commonly been done. uh Have you guys got a prototype like that working.

B

Yes, uh yeah we do so. um The the thing is back to your question. Is you know we use kubocado to register the device right.

E

B

All the metadata is stored here and here right. So then, when the edge node pull down the images, it knows what to pull down. Okay, what kind of like architecture on like there's so.

D

Is it using docker pole though, or is it doing a different kind of pull down.

B

uh It's using similar to docker like the cr and basically we use this oci. We follow the oci protocol.

C

And then at the central, then, can the central still modif monitor what images are running and uh pull off, say rolling, updates and things like that.

B

uh So for the rolling update we haven't tried yet, but uh you know like a from a monitoring perspective using couple, people can just do what we are currently be able to do. Like you query the know the status or part of that status, so the philosophy or principle we are trying to maintain is like even at the edge scenario, we are working as a like. A normal cluster. Things are still continue. Working.

C

Can I ask you, is this open source then? If so, could I get a link to the repo.

B

So uh that's a way. We are we plan to open the source, but uh you know like we are still working on it, so the target is like uh for the kubercon. We will then to announce it and share the well.

C

I don't care if you're still working on it, if you are going to open source and I'll, I'm curious to look at it now, whether it works really.

B

Can I know who's talking.

C

uh Steve wong from vmware.

B

D

And this is preston from google. I think a number of us on this call would be you know, as as the edge working group folks here, who kind of show up regularly. We'd all love to see it. I see, there's a cube edge, org and repo on github. That seems like is that the placeholder for it.

B

Yes, yeah and we are not open, we haven't opened it yet.

D

Right right, I just see it's kind of a like a you know, putting the flag in the ground, yeah yeah um yeah. I mean, I think, there's a lot of uh you know, there's a lot of meat in what we could discuss around like the nature of the way that controller is representing the resource and how the.

B

D

The container content is pulled things like steven mentioned around. Like do you use, I guess the biggest overall question is sort of how and where have you needed to deviate from what kubernetes does as sort of default concepts in order to achieve the functional goals you needed and were there are there any of those places that could, you know essentially be turned into features in kubernetes itself, rather than kind of custom adjacent code?

D

um That's that's kind of my intriguing overall question.

B

I think that's an excellent point uh personally from a pod, spec or conflict map or secret all those kind of thing perspective. Well, I'm implementing the edge controller. I don't see kubernetes lacking any capabilities, so uh the only challenge is around the service. Standpoint, concept, controversies and stuff like that.

D

B

We you know like in the address scenario, as I mentioned, the edge node can be in the private network. How can we ensure that bi-directional network communication is a it's the biggest challenge and then that's why we enable the bus and then, on top of that, how can we enable the service, endpoint service and allow customers to to like just deploy application, then use service to enable the.

A

B

Disk recovery- that's uh one thing I don't see, I think, kubernetes some features. We need to do.

E

Can you talk a little bit more about the bus yeah? That was my question.

C

Yeah or in this diagram it looks like there's: two buses are those independent networks that.

D

Or is this like essentially like a tunnel like a reverse tunnel, so that the edge can communicate with the master as if it were.

B

So those are two services running one on the edge and one on the cloud and then like we follow the the I forgot, the exam, the network uh layers like a level two three and four or seven. They reim implemented each level and then ensure the bi-directional network connection.

B

So the the point is to enable a multiplex with a single connection between one node and a cloud. It can be multiplexed and then bi-directional can.

D

You I mean, is there who can you say anything more like about the specifics of that the networking layers I mean? Is it or is there another? Maybe we can.

B

Maybe we can like a book this uh and then we can talk in detail next, because I'm personally I I like we work as a. We have a team here right, I own the edge controller and uh one of my colleagues or two they work on the bus design. So, okay, let's say you have.

C

A hundred sites are, are there a hundred different q buses, edge edge networks.

B

So I'm not sure if I get your question so the the point is on the clock cloud. There's only one coupe bus uh pro service or like it can be scaled. Okay,.

C

And then edges like, if I have more than your diagram, shows one edge location. But if I had is that two cube bus at edges, or is it one that where it's a a network where they all see each other on the same subnet, no.

B

No, so for the for the multiple action, node scenario, each add node will be the same like they will. Each has a a cool bus, a service, okay,.

D

I got it thanks, yeah and I know maybe maybe feel free again to deflect this until we have the person who maybe can answer but from your understanding is the is the idea of the bust to provide um something that is at the kind of logical message. Layer in terms of you know, uh discrete controller messages, or is it to provide something that is more of a network layer like pass-through for um being able to kind of reach arbitrary kind of cluster ips or that kind of thing.

B

uh It's similar, it's a it's a network overlay.

D

Okay, so it does go down to the ability to sort of say I need to reach a yes like a kubernetes service running in the cloud by a cluster ip address, it'll route, that okay, exactly a.

B

A enabled other service discovery and routing.

D

Okay, so it's kind of like a vpn, basically.

C

Yeah to some degree: okay, yes, yeah and then at a higher level in the network stack you envision, providing a service with uh message queuing so that if connectivity to the central is temporarily out, attempts and things up would be queued rather than having to take that on inside applications.

B

So the the the idea is uh either like a queue or we have a storage uh local at the edge and on the cloud right. So in case of offline, it's disconnected once the connection is rebuilt. Then the tools not storage can be synced through the sync service.

D

And um another this is, this is great stuff by the way, so I just have a question: uh the the metastore. What are you guys using as a backing? Is that just another kubernetes service with a number of storage plugins or what is what is that exactly running so.

B

um Initially, when they prototype, we are using etcd.

D

Okay, so it's it's a key value. It's sort of a keyboard.

B

Yeah, it's an adapted, non-structural key value pair.

E

D

You could you could take it? Are you exposing it um from the like edge controller itself? Are you speaking um fcd using client, fcd libraries directly in the edge controller, or do you have a a small shim running as like a meta store service that just relays.

B

Yeah good question so add controllers calling the like similar to the controller kubernetes controllers, uh calling the utcp client apis.

D

Okay, so you don't you don't yet have a point of kind of decoupling injected in there. So preston. Can you help me understand.

D

So the idea here is in the situation stephen's talking about where you have many many of these or you have maybe other places, you're trying to work with metadata about the edges, instead of relying on on ncd for that, if you wanted to swap in some other managed sort of key value system, whether the edge controller simply had a an adjacent, even a sidecar, that sort of wrote key values that it needed to in an abstract way, and that that thing then would use xcbd fcd by default, but could be configured to store those key value pairs in another system in a way that when you switched it, the edge controller didn't care that you switched it right, because it only knows it's speaking kind of its own api contract to the meta score system.

B

So um I think I didn't- maybe I didn't see it very clear so between here and here there's a protocol right. So then, as long as the protocol is insured, then behind the scene, whatever storage, it can be used.

D

uh Okay, so there is a there is a contract kind of.

C

Yes, what you're saying just to be clear that that disc uh shaped picture down in the edge core uh that metastorage edge you're saying that's an it's. That's an instance of etcd.

B

um So it can be circulated against the same thing. So there's a protocol here, people can use flicker light or epcd here like it can be a queue well.

C

There is one difference in that with ncd: you could conceivably make it a cluster so that you make a rolling update or something, whereas with sql lite. I think you'd have to take an outage. Should you ever update it.

B

The thing is: uh okay, so here here's my personal view because uh keep in mind the edge node can be really light. You know like a memory constraint, so then one has to decide what kind of storage you want to use. Yeah.

D

The right, like you, could use very light.

B

Yeah, plus often time like uh the the ha requirement on the edge node is lower than the cloud.

D

Can you give us an idea of some of the the types of things that the.

B

D

Is storing like what are what are some examples of of the kind of data you're persisting to provide and control.

B

So um the store would store, for example, from a kubernetes orchestration perspective. It will store the products back like a customer put through the cook cuddle and also the the node information or the metadata, and then, on the other hand, from the app engine sync back uh then through the controller, it also store uh the the part status and node status, so that, like it, can report back to the kubernetes api server.

D

And you're doing that, even though you have direct potential direct connectivity from the from a cubelet app engine to the master api server itself, what is the like? I'm trying to understand for some of those functions which are normally done just by a node and the cubelet.

D

You have this app engine which is sort of a derivative of the cubelet right, yes, and now, instead of having that communicate over kind of layer, three network directly to the the api server in the cloud, you're kind of relaying in both directions. Some of this some similar information through this sort of alternate path, and I'm trying to I'm trying to get a little bit of better understanding of why?

D

Why? That's um you know what what that's doing differently or better. In that case.

B

Okay, yeah I'd like to hear all your thoughts as well, so the the main point is to enable the loosely couple, because we assume the add nodes uh can be offline and reconnect, uh on the other hand, is for all the data they are metadata right, so uh have some uh like uh duplic redundancy, really think it's fine. uh The the other one I want to say is, like you know, for kubernetes api server, uh for uh you don't have in the in the in the api server in the etcd tree.

B

You don't have a view of for one node. What are the available parts for the specs rather than it's a part spec? What are the nodes so then, like from a performance perspective, we want to restructure the tree in the meta store and then make sure. Then the the efficiency between the cloud and edge.

D

Interesting interesting so and just to clarify like if I were to do a cube, cuddle, uh get nodes and do kind of a listing of nodes from the cloud. I would see all of the edges as well. Yes, yeah.

B

It will behave as what we have in the cloud.

C

Have you thought of a scenario where, for availability and redundancy you could have an edge location, potentially have talk to multiple masters or fail over to a different one?.

B

um I think that's a good question, so I steve, I would assume you're thinking of when the edge nodes are large scale. Then or like you, as you mentioned, you, you you're thinking about the edges scenario, uh the the you want to because the master one of the masters.

B

From a cloud perspective, uh we plan to enable multiple, aha capabilities as well, just like what kubernetes is enabling right now.

C

Yeah, I just bring it up because, in the context of legacy things like data systems that were iot before they called it, iot uh in high value things like pipeline controls. It's not.

C

B

uh I'm not sure if only me or not you're, no.

D

I'm hearing it too, I don't know whose.

B

D

D

B

So I I think the other one is uh uh conveyor: okay, rephrase or repeat what we talked earlier. I think uh prison asked me about the what uh features uh kubernetes is lacking uh after we enable this architecture. So we talked about the the kubernetes service and then steve asked about the the image repository for different architectures.

B

Maybe others you guys can help me understand what exactly or maybe I didn't follow what his question about, because from a agile perspective, when the edge node configuration is, is clear, then it knows what to pull okay and then I think the third one is about h a uh dj. Do you know if there are other comments, we can write down.

A

Teach us one question about the the: is there any uh special protocol between the edge controller and sync service on on some higher level.

B

uh So, between the edge node and add controller is, as I mentioned, is the regular uh http or tcp connection.

A

Yeah on the on the q bus level, but the controller and sync service are just using http to communicate with each other.

D

A

D

Is it maybe related to jean's question here is: is the idea that the sync service is essentially sinking that edge's slice of the metastore right so holding a branch of the tree foreign particular nodes uh metadata? It gets synced only that edge yeah, yeah.

B

D

That's the primary job of the of that arrow between the edge controller and the sync server.

B

D

B

And then it has some performance implication there too.

F

Hey uh cindy, what kind of workloads do you support on the edge? Is it just parts or do you support other workload? Apis.

D

Like deployments and replication controllers right that that all would need a master and a scheduler type of control.

F

Right exactly so, do you run any controllers at all at the edge or it's just the cubelet, making sure that the part is running.

B

uh Just the the lightweight uh coolant.

F

Okay, so when how do you so from the central plane, when you use cubectl to deploy workloads to the edge like? um How do you do you do like a spray like when you create a part? Is there an object that replicates it on all the edges or you target each edge one by one.

B

Yeah we target each one by one, so so.

F

B

Is a scheduler would still uh do the do its thing and then figure out which part belongs to bind to which node right and so.

F

Forth yeah for targeting you use labeling to make sure that the workloads land on a particular heads- okay, okay, yeah.

B

All right, okay, yeah any like a kubernetes scheduling mechanism, can be, I think, can be extended on this. Based on this structure,.

F

Okay, so um I'm probably curious as to why you chose this design versus running a full controller on the edge just for a lightweight.

B

You mean uh you mean, run the edge controller fully on the edge cord.

F

Like running a kubernetes api on the edge.

D

Okay resource pretty much right like.

F

It starts to get a lot of schedule.

D

To do the api server and the scheduler and all the other control planes, okay,.

C

Okay, well, not only that, I think in this picture, there's one api server at the central if right and that.

F

Means if the connection to the central.

C

Is down if you try to run rollers, which would usually be demanding use of the api they go down.

B

So if I hear your question correctly, you're saying why not have a kubernetes cluster locally on the edge.

F

No, I was just trying yeah. That is correct. I was just trying to understand. uh Had.

B

You looked at that approach and then.

F

B

So the mainly I think, from a very high level, is how we view the edge cloud we see this edge is a resource. It's an extension of the cloud right.

F

So, okay yeah right. So what is the order of like you're looking at out of thousand or order of ten thousand or like 100k edges, supported by a single api.

B

So for that, for that well, we are currently doing the perf evaluation. I don't have the number yet.

F

Okay, uh okay, that's fine! Okay, yeah.

B

Yeah and that's pretty much what I have, I think we can talk more and then I would really like your uh come like feedbacks here. I think uh I heard about uh steve and pristin mentioned about the storage uh using edcp or q or sqlite. uh Can you help me understand like what's the reason you ask that.

D

It's actually probably related to barney's question on the scale of the number of nodes and whether or not that, because at cd's used so much for other key kind of cluster functions in the cloud. If there was.

B

E

D

Point enough scaling out of the number of nodes that that.

E

D

The edge controller and the metastore could be removed to a different key value system to to sort of preserve the the load or function of atd for other. You.

B

Know controlling.

D

Usages and that's hypothetical, I'm not saying that would become an issue. I just think that's that's a potential area where you would want to divide the effort, but you know honestly, it could just be it's an independent fcd instance as well. It just sort of has to be what's mixing the the task on on one ltd. Okay,.

C

I think almost certainly look at people who've used that cd for collateral that once you get a certain sk scale, it ends up being an independent nct instance, and the idea as it gets bad big, can have a pretty big appetite for resources itself, as well as low low latency networking.

B

C

I mean I, I suspect you did it with an abstraction layer such that fcd could be swapped out anyway, so maybe th these don't matter.

C

That was a great presentation and if you're going to be presenting at cubicon china, I um there and love to learn more okay.

E

C

I feel some kind of a birds of a feather. You know open discussion meeting while we're there as well.

B

D

Yeah and I know there's there's um you know legal and all sorts of process reasons, but if there is a there are way for some of us just in this working group to get a preview on.

B

D

um I know several of us would would love to kind of answer more questions ourselves just by looking right.

B

D

More questions, but I don't okay. I could probably answer them if I just looked at code rather than.

B

Okay sounds good, um I I think for that. uh I need to circulate back and then I need to ask yeah and see.

D

A

Maybe in the meantime we can use like and and the forum we have to you know, discuss all the things interested to this topic.

B

Okay yeah great, thank you.

A

Thanks, india yeah. I also think this was this was great and, and we ate up the whole hour for the meeting, so maybe I don't know: if what do you guys think? Maybe we can just leave it here for now.

D

Yeah I'll just say that I'm sorry I had uh book time to do the stuff. I said I was going to do on the white paper and that time on my calendar got now as you all are familiar with. Sometimes things get bumped by other yeah yeah sure I didn't.

B

D

Done what I I was hoping to so apologies. Basically.

A

No worries yeah. I I plan to do more than I did so. Basically, yeah try, try it for the next sprint. Next, two weeks to to to get get document more in shape is what's on my list.

D

Is is any of you attending the mac congress in berlin next week? Is the etsy mech congress.

F

B

E

B

A

Okay yeah, so let's continue chatting online until the next meeting.