Cloud Native Computing Foundation Kubernetes on Edge Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Take the Edge to the Cloud with OpenYurt and EdgeX Foundry - Yixing Jia & Chao Zheng

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Take the Edge to the Cloud with OpenYurt and EdgeX Foundry - Yixing Jia, VMware & Chao Zheng, Alibaba Cloud

According to the application-container market report, Edge applications can benefit a lot from using the application container model. Kubernetes(K8S) has been proven effective in managing containers on the cloud. However, the edge applications are usually fragmented, i.e., various protocols, hardware, etc., making it challenging to use K8S in the edge environment. As a plug-and-play Open Source IoT Edge platform, EdgeX Foundry(EdgeX) allows devices belonging to different manufacturers with different protocols to be connected. On the other hand, OpenYurt extends the native K8S to the edge using only plugins, which supports all upstream K8S features. In this talk, we will present our edge-cloud solution that integrates EdgeX into OpenYurt. Our solution leverages EdgeX’s ability to manage a rich collection of edge devices and uses OpenYurt to manage servers in the edge environment.

A

Hello, everyone- I am charles software engineer at alibaba cloud today we are going to present our work, take the edge to the cloud with open yard and the ajax foundry. This is a cooperation project between alibaba cloud and the vmware.

A

This talk will be divided into four sections. First, I will introduce the four main challenges of applying kubernetes to edge computing environment and how open yard address these four challenges, then eachine will provide the necessary backgrounds of ajax foundry and how we combine open yard with ajax foundry.

A

And finally, we will wrap up the talk with a demo.

A

So why we need to use open yard and why we cannot use kubernetes in edge computing environment directly. To answer this question, we have to understand that applying kubernetes to the edge computing environment requiring us to overcome four challenges.

A

First, how can we prevent the parts from being evicted if edge nodes are temporarily disconnected from the api server? Second, how can api servers send requests to an edge node if they are located in two isolated networks?

A

Third, how can we deploy workload by region as system settings or device models in different regions may vary four? How can we manage edge device with different protocols in a kubernetes native way?

A

We develop open yard to address these four challenges. The design principle of open yard is extend the native kubernetes to the edge. Therefore, we won't change the system architecture or any core components of the kubernetes.

A

In the other words, we will tackle all the four challenges only using the native kubernetes workloads. In addition, we always keep the goal in mind that we have to support all upstream kubernetes features.

A

Here is the system architecture of open yard, as we can see to have an open yard cluster up and running. We only need to deploy several controllers on cloud nodes and on each edge node. We only need to deploy yard harp and the yard tunnel agent.

A

In addition, we provide a command line tool yard ctl to help us to manage an open yard cluster. By simply running iota, convert all the above components will be automatically deployed.

A

Next, I will introduce each of these components and how they help us to tackle the four challenges.

A

The first problem we try to solve is: how can we prevent parts from being evicted when a network connection between an edge node and an api server has been cut off in an edge computing environment? The network connection between edge nodes and api server can be unstable.

A

However, kubernetes require the api server to be tidally connected to worker nodes. If a worker node is disconnected from the api server or the api server misses several heartbeats from the worker node due to poor network connection, the node will be marked as unready and the parts running on the node will be erased from atcd later, when a worker node reconnects to the api server, the kubelet will notice that parts running on it are not in the etcd.

A

Then it will physically evict the parts. This is something we don't want it to happen.

A

To avoid this from happening, we develop yacht hub yacht hub functions as a reverse proxy and local cache that is deployed on each edge node after it is up and running. We will connect the kubelets to the yacht hub instead of connecting to the api server on the cloud during the runtime. The yacht hub will keep checking the healthiness of the network connection between the edge node and an api server.

A

If the network between the edge node and api server is stable, kubelet will talk to the api server directly and synchronize the pass state, as it does in the native kubernetes in the meanwhile, the yacht hub will cache post states in the local cache storage.

A

If the network between the edge node and api server is cut off, the kubelet will use pass states in a local storage instead in.

B

A

We have a garbage collector running in github, which periodically removes useless pod stage from the local cache. At the same time, we have the yacht controller manager running on the cloud to prohibit api server from expelling parts from unready edge nodes.

A

The second problem we wanted to resolve is: how can we redirect the request from api server to edge nodes? It is very common that users may want to fetch logs of pods or execute commands in parts.

A

This operation are normally conducted by using the kubictl command line tool, which will send http request to the api server first and then api server will redirect the request to corresponding nodes. However, in a production environment, api server is usually located on the cloud while edge nodes are located in user's private local networks.

A

Therefore, api server cannot talk to edge node directly. We developed the yacht tunnel to tackle this problem yacht tunnel using a client server architecture on each cloud. Node we deploy a yacht tunnel server while on each edge node a yacht tunnel agent will be deployed even though the incoming traffic may be blocked outside of the private local network.

A

Edge nodes can still send requests out of the local network. Therefore, after yacht tunnel agent is set up, it will initiate a reverse connection and register itself on the yacht tunnel server to redirect the request to natano server. We inject the rules to the host ip table, which will redirect all outgoing http requests with destination part 10250 and 10255 to the tunnel.

A

Server yeah tunnel used the api server network proxy internally, I will refer to api server network proxy as an amp in the rest of this talk, amp does not work with old version of kubernetes that does not support the egress selector.

A

Therefore, we add an interceptor inside a tunnel server which will encapsulate http request in a format that is compatible with amp.

A

Also, since the amp used grpc protocol to transfer data between tunnel server and a tunnel agent, we are able to further reduce the bandwidth between the cloud nodes and edge nodes after a http request. Is encapsulated, the amp server will pick the corresponding amp agent and send a request to it.

A

The third challenge that prevents using kubernetes in edge computing environment is that kubernetes does not support deployed workloads by regions since an edge cluster can be located across multiple network or geographic regions.

A

An assistant or device setting in different regions may vary, so users may want to deploy workloads by region, for example the system architecture of edge nodes in a region 1, maybe amd64, while in region 2, maybe arm64.

A

If users want to run same workload in both regions, then they have to create two workloads with exactly the same configurations, but only difference in container image name. This can cause exponential increase in the maintenance efforts. With the increasing number of workloads we developed the yacht, app manager to add the maintenance burden on users.

A

Specifically, the yacht app manager contains two controllers: the no pool controller and the united deployment controller.

A

The no pool controller can group nodes into pools can be nodes with same system architecture or nodes from the same network region, and then we can manage nodes in a sample uniformly.

A

For example, we can add labels annotations or tense to all nodes in a sample after we grouping nodes into pool united deployment controller can deploy the workloads based on a send template, but different configurations across pools. In addition, we leverage the service topology feature to bound east western network traffic within a pool.

A

The last challenge is: how can we support various kinds of edge device made by different manufacturers using different communication protocols?

A

Existing solutions either need to change some key components of the kubernetes significantly or require users to implement the device adapter or driver from scratch, which result in large development effort and the laws of some upstream features.

A

Inspired by the unix philosophy. Do one thing and do it well. We believe that kubernetes should focus on managing computing nodes, while the device management should be down by a mature edge computing platform.

A

Therefore, we come up with the idea of integrating edge x, foundry into open yard. Next eaching will give us some necessary backgrounds of ajax foundry and introduce how we combine rgx foundry with openyard.

B

Okay, thanks charles for the introduction, hello, everyone, I'm ishin from vmware city office and also the maintainer of the open europe project.

B

So for those who are not familiar with edge, foundry I'd I'd like to give a general introduction first, then I will cover the open europe and ajax integration overview. At last. There will be a demo to illustrate how it works.

B

So what is adjacent foundry? It is an open source when the neutron project and the linux foundation with apigee 2 license. It is also microservice loosely coupled software framework for iot edge computing and at the same time, it is hardware and os agnostic.

B

If we need to use one sentence to describe it. uh Ajax is a middleware that connects things with your I.t environment. So clearly speaking,.

B

The surface of edx constitute a dual transformation: engine first is translating information coming from sensors and the devices vary hundreds of protocols and a thousand of from mice into hx, and second, it can deliver data to applications, enterprise and cloud system over tcp based protocol information and the structure of customer choice.

B

Now, let's take a look at the edx architecture from south side to the north side. It have. It has four layers: the device surface layer which is responsible to connect all kinds of sensor devices or activators over hundreds of ot protocol, the co-service layer, the supporting service layer and the application service layer.

B

Let's take the data flow in edx system, for example, to illustrate how it works.

B

First sensor data is clacked by the device service from a scene, and the data is passed to the code service for local persistence.

B

Data is then passed to application service for transformation from my team featuring and they can zoom back into mouse to enterprise and cloud system when needed.

B

Data is then available for ad analysis and can trigger device activation through command service. Basically, there are two parts to control the added device. One is through the cloud. The other is some automation, control and add through some predefined logic, through the user defined application service or through engine or both. Since you can always combine them together, like you can training models on the cloud and the creator and application service on the edge for influence.

B

Now, let's take a look at the edx deployment scenario in today's iot landscape, it is imperative to leverage computer storage and network resource wherever they live. So in the first scenario you can deploy the device service on the edge side and everything else in the cloud.

B

This is for latency in sensitive applications and the second scenario, if the gateway has enough compute and storage resource, you can deploy the full idx foundry service on the gateway and gain good response time in scenario 3. If the gateway is not set good, you can just deploy the device service and the core service there and move the application and the analytics to the server part. The fourth scenario is quite similar with the first one, except that it deploy ajax on the phone server to gain better latency response.

B

So we can see that the quantity and the function of macroservice deployed on given node depends on the use cases and the capability of the hardware.

B

Now, let's take a look at the open unit and the adjacent foundry integration on the cloud side, we run the use, control manager, you're, the app manager and the tunnel server manager.

B

You can join it on any public, private or hyper cloud or on parameter center, which, according to your requirements and on the other side, besides the default, you have an internal agent.

B

We will deploy a new device controller which is responsible for sync data between the hx and the kubernetes, so the xcode service will also be deployed on the edge side, in some case that sensor's direct connect, githui doesn't have enough resource. We can just deploy the edx device service there. The add node can be either unbased or x86 based.

B

Now I'll use a demo to illustrate how it works.

B

Let's take a look at the demo environment topology. We have a mass node running on the cloud and we'll have two work nodes running in the private network.

B

The python is in my home office and the e-box is running in another location. They together compose a unified, coordinate cluster cross cloud and edge with different hardware architecture.

B

We use kubcontrol to get a node to check the node status in the cluster. This is the pi volt node. We connect the temperature and the hemi sensors through gpl, pin.

B

17., this is an rgb led light connected to the pi through cpl pin, 18, 19 and.

B

B

This is the second node which is x86 based.

B

We next will deploy the ajax code service to the pi forward. Node.

B

We use kubercontrol getport to check the deployment creating.

B

Status after few minutes, we run the command again.

B

We can see that the idx service are up and running on the pi forward. Node the device controller will sink the device information to create.

B

We can use control, get the device to list all the devices in the azx. We can use group control, get device profile to list all the device profile information.

B

Also, we can use google control device service to list the device service information like what we can do to the traditional kubernetes object. We can change the device status by group control edit.

B

First, we will run command group control. I id device name.

B

We then set the device property light to color right as the desired value we save and.

B

Exit you can see as x will turn the led light to right.

B

And the next we run control. I t device name again.

B

This time we'll set the desired color to blue.

B

We save and exit we can see. Ajax turns the light to blue last. We run group control, id's device name again and cite the desired color to.

B

Green then we save and exit we can see. Ajax will return the led light to green. So with hx and open urls, you can manage your id devices in your kubernetes native manner, thanks for listening our session now, it's clear and a.

B