Cloud Native Computing Foundation Kubernetes on Edge Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Robotics Edge Cluster System - Tomoya Fujita, Sony Corporation & Feng Gao, Sony China Ltd

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes Robotics Edge Cluster System - Tomoya Fujita, Sony Corporation & Feng Gao, Sony China Limited

Sony R&D Center has been though robotics history and products for years. As robotics platform and Robotics Operating System (ROS) getting matured, there is a requirement to handle the distributed system integration. Using Kubernetes on edge cluster system, there are a lot of advantages such as application lifecycle, deployment and recovery. Also using CNI and ROS Data Distributed System, it can construct distributed system on edge cluster, so that multiple robots can connect directedly and work collaboratively for the specific task. We will share how we can use Kubernetes on edge including deployment robotics application and possible problems based on our experience. Furthermore, we will share our approach to support edge dependent platform with device-plugin to attach hardware resources and even virtual devices which access to the host system such as 3rd party application.

A

Hi welcome to cuban this age day. I am thomas fujita with sony cooperation and we have a co-speaker from sony, china. He is also software engineer named fengal. First of all, we really appreciate for this opportunity and a community effort. Thanks for having us and today we're going to talk about kubernetes, roblex edge class system, I will be sharing the most of the overviews and the rest will be taken care by fan.

A

Here is the today's agenda for this presentation, starting with the introduction we will go through the background, the problems and the requirements, especially for hilt, use cases, and we will talk about what we want to achieve with using kubernetes as in architecture with distributed system, then we can even go deeper about roblox examples with kubernetes to support distributed system on edge platform.

A

So I am, my name: is tom fusda with sony randy center at sony corporation? I am software architect and developer, we're most likely working on system services. Middleware, like loss equipment, is in open source aspect, and I am also a member of uh robot operating system. Technical stealing committee.

B

I'm a phone with sony, china responsible for software such as kubernetes and the multimedia framework.

A

Okay, so please feel free to reach out to us anytime. Besides, we are going to be, we will be available on the slack channel or g iot dash edge.

A

So let me do quick introduction about sony's purpose. The sony's purpose is to fill the world with emotion through the power of creativity and technology. Our foundation is based on technology to create the new values within diverse and many businesses, such as data center data content, hardware, sensors, consumer devices, medical and financial services.

A

Okay, let's get started at first I'd like to come up with some backgrounds in a couple of slides. As we know, it's been happening that each device is getting matured with computation connectivity and hardware, accelerations and so on. At the same time, each system gets complicated to support the distributed system and connected system.

A

Besides, concerning about perception, recognition with dynamic sensing data on the edge, the application should be dynamically adjusted against the environment. That is said. The system should be like circular faction system to adjust the dynamic environment.

A

So what comes to roblox and robot use cases as background the robots are expected to work and help with highly tasks even with multiple robots working together. This is easy to see if we can think about factory logistics, rescue and entertainment use cases, and if that is single robot that we are dealing with, there should be no problem to do operation, but what, if it comes to like 10, 100 or thousands of robots, we have to control and maintain?

A

We surely need to consider that cases to do our development and maintenance easy without any operation as well as possible, especially during development. The application developer does not want to do any operation to check the application. It has to be easy, quick and efficient to go.

A

Importantly on its devices, the problem becomes complicated and there is a lot of dependencies compared to the cloud infrastructure think about the application portability modularity. There should be some abstraction layer to conceal the hardware devices.

A

From next two slides, let me introduce quickly about robot operating system, which is called ross as aspect from kubernetes. We can just say ross is one of the runtime framework and sdk.

A

The robot operating system is a set of software libraries and tools that help you build your robot application from drivers to state-of-the-art algos and with the powerful developer tools. Ros has what you need for your next roblex project and it's all open source.

A

What ros provides is not only for sdk but also simulation tools which are really important and useful for robot application. Development.

A

The gazebo is the simulation tool constructed on physics, sensors and interfaces, including dui. You can switch real world and simulation world easily, and application is agnostic from this world. That is said you can just go, develop the application without any hardware, but simulation at the very first place using ros and gazebo. You can have everything you need to develop your robotics application.

A

This ros overview is needed for explanation and you can find a way more information about ross on the internet.

A

So now we stop here about talking about ross and let's be back on the main topic, which is kubernetes disrelated.

A

Okay, so let's move on the topics now to start with the problems. What is the pain, as you can see here, is the current destruction that we have. The application needs to be integrated into the specific system every single time, even if the application functionality is almost the same, there will be some operation cost for integration. This is because platform and system is different from one to another in the edge and it takes time for application to system developers.

A

What's the huge pain here is that is real, that is just the operation cost, but not exactly the development. We do not doing that. We do not like doing that anymore. We think nobody does.

A

So we are proposing this cloud and edge common architecture. It is simple and common, it also can support distributed system and the application should be platform agnostic, in other words, against the pain from the previous slide.

A

Once we get application developed, we can run that application wherever we like could be cloud could be edge, could be you don't need to even care. Besides, you can see blue highlighted directions in the image on the right top in the edge network, devices are connected, connect connecting each other directly as distributed system to keep the performers.

A

Continue to the previous slides, the application engineers aspect, it is just appears to be a single entry point to manage the application.

A

We do not need to operate for each devices anymore, but just accessing the dashboard front end will be all sometimes we want to deploy application only on specific debug host, sometimes all of the nodes introducing the boundary between system and the application application just requests. What kind of capability or hardware devices are connected to run the application so that the rest will be taken care by framework and the application can be agnostic from the platform or dependencies.

A

This is exactly one of the x system in the edge world. That is something we want to.

A

Achieve so, we have been considering kubernetes against this situation and the problems described before and the answer. For now. We could do that with kubernetes with kubernetes. We can have most of our requirement, as described as before, such as development and application maintenance, roll up and down the application without any downtime. We can also keep the application running administration and device capability management and scalability up to ten thousand nodes in the cluster.

A

Those are everything that could bend. This main line provides and allows us to do that as it is.

A

So, after all taking advantage of kubernetes, we can actually support this architecture. We can have the flexible cloud and edge cluster system for application, development and management and providing security, enclaves and capabilities dynamically attached to the runtime pod when application starts running.

A

This is what exactly we want to support as cloud and as common architecture from the next slides fan will be taking over the presentation to explain more details about its specific use cases and what we have done with kubernetes.

B

Thanks I will be taking over from here. We have we shared enough about overviews and the requirements. So let's talk about more details here we are going to talk about the four main subjects: dynamic cluster reconfiguration for hiot devices, the system and the application will rules. Dynamic security includes attachment with resource and the hardware about structuring with the kubernetes device, plugin interface and the implementation.

B

Here is the describer I distributed the system on edge environment as in a typical use case. Expecting multiple robots are connected in the sim lab and working together for user application is built on top of ros, which is publisher, subscriber architectures, as application layer to support the distributed system.

B

There are phase detection, ad detection container runs inside ports on the worker nodes, and the selector container can select the images to notify the visualizer to display what image should be printing the mid monitor on primary node. This is one of the example, but we can do this with kubernetes, since this is a distributed system, it feels independently and it appears to be a single system, as user experience.

B

The point what we want to mention here on this slide is: we need to use willnet cni. This will be needed if the application layer use multicast. Since the gloss supports distributed system, it does endpoint the discovery at the runtime with multicast.

B

We have tried a few other cni's, but we'll work us out of the box when we use source.

B

Thinking about consumer devices it distributed system within lines and even with third-party application, they are shown or must be securities certificate or key to control author resisting and access permission for each endpoint security turns to be considered after development, but once it comes to production phase that something we cannot just ignore here, it describes how to manage security includes as administrator and the user and the whole security includes are attached to appropriate ports dynamically.

B

This is just something we can do with kubernetes custom resource named configure map and the secrets first administrator rejects the security and clues for each endpoint and give them appropriate access permission where api server, so that we can control the access permission in the first place.

B

For example, third-party application developer can only see a couple of security includes for third party application, and when we need to run the application, we can just see what we need to use for this port as security enclosed. The rest will be taken care by kubernetes.

B

It will check the permission if the user is allowed to use that security, enclave and zoom loads and attach the required security include dynamically.

B

On the endpoint port, as volatile storage on the physical machine application, ports can be agnostic from this binding by kubernetes, but it just uses security includes to participate, distribute the system and access the data objects once ports are shut down, security and clues will be gone too. So far we confirmed that everything works. Okay with those security features like this.

B

In edge devices, one of the most complication is platform dependency. There are so many devices we cannot even count all, but this should be also abstracted to application perspective.

B

Kubernetes has interface named device plugin, which is expected to use for gpu originally, as described here us plug-in.

B

Implementation can be one of the kubernetes custom resource and the interface are really simplified, so that it should be easy to support the implementation device, plugin iterates with kubernetes to list a locate and banned the device for application ports dynamically. This really seemed to be kind of a strategy that we want to have in edge use cases. So we did try to use that against our requirements.

B

In ig iot devices there would be more complicated devices such as fpga, special hardware, acceleration dsp and so on. Also, we would want to add a virtual device provided as api exercise to the host system, instead of finding physical devices directly to application port providing api from host system to application container.

B

That gives us more flexibility and security to control access from container host system. We already have the implementation something described on this slide. Application can request the device api if it has read access when application container starts internally or device plugin iterates with the host system manager and check the permission and provide the api is requested by application.

B

And the device apis are attached to application container dynamically. This entire process of device plugin is perfectly concealed by application aspect. So for application, it is agnostic from this process and just goes with device or apis.

B

So do we have everything we need for device plugin? The answer is no. There is a open issue for device plugin, which originally comes from our use case. As we mentioned before, ig devices are way more complicated. Some devices are not simple: to manage with open close button repair more and some specific operation will release the resource. Us plugin currently does not have such kind of callback interface to release the device resource, so we have been working on with the community to support this requirement in the mainland.

B

uh If you are interested in this, please take a look at the kep and leave some feedback on the comments.

B

Finally, we are gonna talk about the dynamic cluster reconfiguration, as you can imagine, ig environment. There are some situations different from cloud infrastructure at the first robot moves, weak moves. That means accidentally robust moves out of the network and the url, where this network is used in id devices which we can see unstable network. In addition, ig devices can be easily shut down or powered off, including this operation and works with battery breaks down easily, and the cost should be well considered going with this situation and environment to use cluster system edge.

B

It has to be robust and reconfigurable without manual operation. Some methods are provided, such as the high availability by the way say it should be more dynamically reconfigured. Only at us.

B

We expected this requirement for cluster reconfiguration in edge device. As you can see the picture on the left. There are some candidates for primary service. Technically, all devices can be primary, energy will do leader election within candidates and once primary is online, that service is notified to all the worker nodes.

B

Even candid nodes become work nodes as well. If the workload comes online cluster network, it will detect the service is available, dynamically and participate in the cluster system. West west also name space should be applied. Thinking about that use case, multiple cluster network in the streamline for factory and the logistics.

B

This is needed to support. So far, we have been developing this framework based on kubernetes api, to see if what's more missing for all use cases.

B

There are just the ideas, but we expect that this is also needed to support as a total solution for h-cluster system. In the future, application redeployment is required, unmore, dynamic sensing date such as location, perception and recognition date, for example, if the location changes for physical world application will be redeployed to adjust to that location, robots in hospitals should be aware of medical context.

B

Robots in the store should be aware of food drink context. Under support distributed system. We will need a kind of set architecture to detect the failure failure in entire system, without that it will be so hard to maintain the entire cluster system in ig devices.

B

At this point, we think we can learn and study more from cloud experience and probably supporting microcontrollers with cube edge, more lightweight agent compared to cubelet supporting this requirement. We believe that we can actually support kubernetes clustering edge system.

B

uh That is all of our presentation, appreciate for watching all this, and if you are interested, please feel free to make contact with us. Thanks.