Cloud Native Computing Foundation Kubernetes on Edge Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ready, Set, Go for K8s at edge today for AI/ML with Tensorflow - Amol Chobe & Sachin Rathee, Red Hat

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Ready, Set, Go for K8s at edge today for AI/ML with Tensorflow - Amol Chobe & Sachin Rathee, Red Hat

At the edge, you typically have to work with a smaller footprint of infrastructure that don’t have enough capacity to run K8s effectively, such as with the majority of IoT devices that are just sensors. Considerations should also be made for bandwidth, connectivity and latency issues at edge. We will discuss potential K8s architecture patterns that can be used for edge workloads, along with all the different elements you’ll need to build an architecture that matches each application’s requirement of low-latency, data privacy and bandwidth scalability. With TensorFlow running at the edge within K8s, you can deploy deep learning models as microservices. We discuss how to apply ML models to detect real-time IoT anomaly and threat mitigation.

A

Good morning, good afternoon and good evening welcome to kubernetes at hd. We hope everyone is doing well in these interesting times. Today we are presenting on kubernetes at edge for aiml solutions. My name is sachin rati, I'm a principal for edge cloud solutions at red hat.

B

Hi, my name is amul chobay. I'm a solution. Architect lead at red hat. So let's go over the agenda at the part of today's presentation, we'll go over following topics we'll cover what is aid to understand the definition of aid? What are typical challenges at age deployment, architecture, type for kubernetes at 8, we'll go into machine learning and understand what is tensorflow what technology we have used in our edge solution design, and then we will go over the architecture for iot anomaly detection using tensorflow.

A

So, let's start by understanding what is edge now edge is defined as any computing and network resource along the path between data sources and the cloud data centers, so edge is not a point. Rather, edges are continuum, as you can see on this picture, the edge spans from edge devices all the way back to the centralized cloud and in between you can have multiple intermediate locations which can host your additional edge infrastructure.

A

The location of your edge application and infrastructure is ultimately going to be defined by a number of factors such as latency response times available, compute network or storage resources.

B

So eight computing challenges some of the key challenges for adopting eight h computing are listed here when we look at edge computing. There are a lot of different choices of hardware and we need to define a solution which will cater this large landscape of hardware.

B

So when it comes to data locality and security, the rules for governing the data changes based on locality of data, and this brings in the compliance as well as security issue, along with the privacy law. The confidentiality of data can often be at rest and poses a huge challenge.

B

Eight devices are resource, constrained and energy sensitive and it's add a huge challenge and workload. Priority become important. Another aspect is maintenance. Device maintenance is one of the key challenge, for example, how I will update the device or packet reward bandwidth challenges, so balance challenges arrived due to multiple workloads sharing the same network part. So we have to be creative about this and use workload, prioritization or network policy restriction to overcome this pump also due to unreliable nature of network, each node need to be autonomously managed itself, and these challenges can be overcome using kubernetes.

B

Kubernetes has different ways to navigate around it, or it can be overcome using uh other its solutions that we can come up with.

A

So with that understanding of the challenges now, let's take a look at three different deployment models for containers at the edge first and most straightforward option for deployment includes deploying kubernetes all in one cluster. This provides capabilities to have a lightweight kubernetes orchestration, where resources are minimal and response times are quite stringent.

A

Second option is where we continue to have stringent response time requirements, but we have a bit more resources which, however, may be distributed at multiple locations. Here we separate out the kubernetes control plane from the data plane, control plane becomes centralized, while worker nodes are distributed to multiple locations.

A

Finally, we're going to have requirements where we see all containers already deployed in the devices. In this case, the management of those devices gets taken care of directly by another linux foundation, project that we have been recently looking at called open horizon.

B

Now, let's get into the machine learning aspect of this presentation, so let's understand what is tensorflow, the definition is written there, but typically the tensorflow is machine learning framework and it might be your new best friend. If you have a lot of data and you are after state of art solution in artificial intelligence like deep learning or neural network tensorflow has been used in voice recognition, text-based application, image, recognition, time, series, algorithm, video detection and many more tensorflow is open source and you can download it for free and get started immediately.

B

You can build and train machine learning models using apis like kerala. You can train and deploy model in cloud on-prem or even on edge. You can see on slide. There is the vast ecosystem of tensorflow deployment type for our project. We use tensorflow code as well as tensorflow lite.

A

In order to build the edge solutions, a number of components come into play, so, firstly, we need a messaging platform. This is needed to accept data or mqtt protocol and massive provides this capability with the apache active mq artemis. Next for real-time streaming, we make use of kafka and kubernetes, and this is provided by project. Strimzy camel k provides the capability for mediation and routing of the sensory data. We also make use of prometheus on the operation side for monitoring and alerting requirements.

A

Sef here is used to provide storage for sensor data coming in for the training of the models, as well as for historical reasons. Jupiter notebook helps in providing the development environment for building and training the model.

A

The developed artifacts are then checked into a source code management system such as git, from where the container images can be built and stored in container registry, such as square.

B

Now, let's get into architecture of iot anomaly detection, so we have used iot device simulator in our design. We have deployed kubernetes at page using all in one cluster deployment model. You can see it on your bottom of your screen. We have also used kubernetes on cloud on the top, which can be run on any public cloud or even on private car.

B

You will see the component mentioned by searching in previous slide in this diagram. Also from kubernetes as age point of view, you will see that we will send the alert to webhook and webhook can be used for feedback loop control as well as alerting the end user.

B

In the next slide. Sachin will go over the details on training model flow for our architecture, design.

A

Excellent, so here the training workflow is highlighted in the red and that's what we're going to follow. The networking data captured on the devices is sent over the mqtt protocol to nmase, which of course, has the artemis broker running.

A

This data is then transformed in camel care and from this point on the training and inference models, kind of diverge. Those are two different workflows from this point for training workflow. This data is passed on to camel k for any normalization and then stored in the cef data store. This allows the models and code running with jupyter notebooks to use this data for training and testing the models.

A

Now, once the model training is done, the artifacts and code are then checked into git from here on, the build process is kicked off and the resulting image is stored in query registry from here it can be replicated to any other registries at any edge locations.

B

So, let's understand the serving model from the tensorflow.

B

If you look at this diagram, the green dots will show you the serving model and we're talking about step number four, where the tensorflow is running on python um application for the serving and prediction the model serving is simply the exposure of the trained model so that it can be accessed by endpoint input here can be direct user or other software. We are using auto encoder model intensive flow. Auto encoder is neural network which takes in-stream data and create output.

B

We are looking at mean squared error to detect anomaly if the real-time data mean squared error is more than threshold, prometheus will scrape the data of top end result and then using prometheus alert manager to notify the end user webhooks can be used as a feedback loop control to remediate the detected anomaly.

B

We have designed and prototyped this solution for anomaly detection. We can use this design in a lot of different scenarios. The consistency and predictability become mission, critical for this edge solution. Again, this is not the only design for anomaly detection, using machine learning for eight devices.

B

And that is all we can cover in allocated. 10 minutes would love to go over our code and design model, hopefully in some other presentation. However, please do reach out to us for any customization of this solution or any specific use case that you may have thanks a lot for attending. Thank you very much so.

A

With that, we sincerely hope you enjoyed this session here. Are our email addresses? Please do let us know how we can help. Also don't forget to provide your feedback enjoy the rest of the sessions at kubernetes hd goodbye.