Cloud Native Computing Foundation Kubernetes on Edge Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Streaming analytics on edge with Kuiper and KubeEdge - Rocky Jin, EMQ Technologies Co., Ltd.

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Streaming analytics on edge with Kuiper and KubeEdge - Rocky Jin, EMQ Technologies Co., Ltd.

EMQ X Kuiper is an open source edge lightweight IoT data analytics/streaming software, and it can be run at all kinds of resource constrained edge devices. KubeEdge is an open source system for extending native containerized application orchestration capabilities to hosts at Edge. The session introduces how to use KubeEdge's application orchestration capabilities and data streaming analytics provided by Kuiper. The session covers Kuiper data ETL, order, group, aggregation, SQL functions, and binary data processing. Also introducing extension for audio and video AI analytic, and the time windows required for IoT analytics. Finally introduce KubeEdge cloud-edge collaboration channel, how user can manage analysis logics at cloud side and accommodate to agile business requirements.

A

Hello, everybody I'm so happy to be here in the coordinates edge today share with you about my thoughts for the energy computing.

A

So my today's topic is streaming analytics on edge with quip and google edge.

A

Before starting the session, I will introduce a little bit about myself, so my name is rocky jin and I'm the initiator of emqx creeper project and also I'm the cooperage iiiot worker group member and also the lfh ex-foundry projector committer, and also the application and the china working group.

A

Member I'm also the former ibm china, software development lab as technology and the product leader and architect.

A

Now, let's look into what's the streaming analytics so streaming analytics? Is our software or framework for stateful computation of unbounded data streams, so user can manage, monitor and analyze real-time of live streaming data, so the typical such kind of software is apache link or apache spark, but both of the software are typically running at the cloud data center.

A

That is not a fit for the edge computing.

A

Why apache, flink and spark are not a field for the edge streaming analytics? It has a it has following three reasons. The first is that the latency, so it will take some time to send the data from the edge to cloud and the secondary is the data security, so it is not fearful or for all of the data sent into the cloud for security reasons, and the last is the banner wise costs.

A

So for some user scenarios the data collected frequency would be very, very high, so it worked. It will cause a lot of banner wise if you want to send all of the data from the edge to cloud, for example, in the internet of a vehicle, or I iot scenario, it will take a lot of bandwidth if collected the data from the edge to cloud.

A

So there are three changes for edge streaming analytics.

A

The first is that lightweight and high efficiency, so for most of scenarios, the device or computer or the server deployed as edge, is not very good, so it has restricted resources.

A

The cpu and the memory is not very good, but we will have such kind of software running at the resources restricted devices and the next is higher and the flexible.

A

So the id change is more frequency than before we needed to have our application or software to support the igor environment and the nasa is deployment and management, so the device are deployed with distribute.

A

They are not centralized as in the cloud and also it possibly with very weak network as success. So the software needed to be deployed to the azure devices.

A

It will take a lot of effort through this.

A

To resolve the issues, so we propose the creeper project. The quiver project is an open source software for iot edge analytics. The project is initiated by emq, so I can introduce a little bit about the emq.

A

The inquir is an open source, iot infrastructure software provide, and now it has a mqtt message broke, which is the most popular open source software project in the github below I list some of the creeper milestones so in october, 2019 the creeper project was open, source and register the first version and in the 2020 we have three major milestones.

A

The first is that we integrated with azure x foundry, and secondly, we integrated with google edge and also in october last year, the first stable major version was released and in the latest version was released at february this year.

A

The creeper project supported the binary data processing and the machine, learning or ai function, support.

A

Here is the overview information of the creep, so I just mentioned in the previous slides, so the projector or the software deployed at the edge must be lightweight.

A

The quiver project is a binary installable and also provide the docker images.

A

Currently, it has only about eight megabytes installed package and the 10 megabytes initial memory overhead, so it is quite lightweight and also it has both different kinds of cpu architect, for example the amd and arm and ppc, and also it supports different kind of linux systems.

A

And open wrt mac os and also provide the docker images below I listed. Some of the performance data that are running at raspberry pi 3, b, plus with the transaction per second, is about twelve thousand and the cpu consumption is about 70 and the memory usage is about 20 megabytes.

A

Of course, the benchmark data is up to the different kind of user scenarios.

A

So for detailed test scenarios, you can refer to the url in the slides.

A

So, by using the you can achieve the goal of the data etl, I mean data extraction, data transformation and the data loading at the edge. So if you look at the right side of the picture, we provide the source and at the right side, is syncs in the middle of the architecture. Diagram is creeper, run time, so, for the data extraction user can specify different kind of data source.

A

It could be the mqtt data source or mq data source, or even the http data source or database data source with the data transformation.

A

So we provide the sql-like language, user can achieve analytics and transformation with sql and the data loading, so user can save the result. Data to different kind of the target system, for example the mqtt the file, the http and also possible, is a all kinds of the database.

A

So here are three steps to use the clip. The first is to create a stream, so our stream is similar to our data source that defined where the data comes from in the right. Here's an example, so the data source is from the topic of an mqtt broke with the data. Center format is a json type and the next step is to specify a rule, so rules better find how to process the data and also where to send out the analysis result.

A

So here we have a sql property, which is specified the sql logic and actually means that, whereas the data or the results without data sender to so the first is that a log which means that the data will be printed to the log file and the second is mqtt so which means that, as a result, will be sent to the broke.emqs.io with the topical device slash without and the last step to submit and then run the rule.

A

So user can issue a command against as a creeper address server with the rules specified in the previous step. So that is all of the three steps to use a creep.

A

Here I will introduce a little more detail for the sql analytics.

A

So first of all, we provided the building functions, which includes a mathematical string, aggregation conversion, encoding, analytic coding, hashing, json, processing and others totally near about 80 functions and also creeper provides a field.

A

So user can filter the data by well or case rank sql class. Also, it provides the join, so user can use different kinds of join. The left join right, join, full, join and across join, so one string can be joined to another string. The string is a dynamic of flowing data and also user can join to another tables.

A

The tables is static data, so it is normally used for associating additional information. So, for example, if the report data has an id but the target system or things needed to get the name, so it can be joined to the tables to get the related name and next is a window. So it provides the tumbling hopping sliding session and account.

A

So the window is often used in the iot testers iot user scenarios.

A

User can use a window to calculate different kind of analytical results, which is specified of a period of time and the last uh provided by the sql is a group buy and order by so user can group by different kind of conditions and other by specified fields.

A

Here introduce about advanced analytics provided by the quip, so creeper country also supported by that the binary data type spot.

A

The first is a binary image, processing, so user can resize or reduce the resolution of the image before sending to the card. So in the right side is an example and also it provides the two hash functions for processing the longitude and the latitude.

A

So user can call the geohesit encode geohedge decode, geohash neighbor and the geocache bonding box, and so on.

A

The last one is that the user also can use machine learning or ai streaming processing, so there are two approach to achieve the goal. The first is that it encapsulates the machine or machine learning or ai function, which creeper plug-in by this approach. User can get better performance, but with higher development and the maintenance effort, and the next approach is to use machine learning, ai services by rpc or rest api.

A

So in some awful user scenarios, the machine learning or ai is exposed with a rest, api or rpc service, so the creepers can support invoke these interfaces with function. Call with this approach, lower development and deployment effort, but sacrifice some performance so at the right side. Here's an example: the input is the image data byte array and we want to label the image.

A

So user can just rather the sequel, such as select label image function from the stream, so the output would be the image content of the image.

A

Next is extension and plug-in, so creeper provides three extension points: source sync and function after extension. Users then can use it in the creeper framework. So here are three steps to achieve the goal or developer appliance.

A

The first is development and debug in your local environment and then compile to the excel file with the c environment as running in the production and then deploy the plugins.

A

So in the live site I listed a source extension example, so user will have two implemented three functions or interfaces that required for the source, plugin, the configuration, the open close and the last is the initial nice instance for your plugin.

A

So this is a native plugin environment development. So advantages is that it can achieve the better performance because the parking is loaded with native, because the disadvantage is that it has very strict.

A

It has very strict limitations, uh which you come from. The gold language, the creeper plugin, is based on the gold land plug-in, so user will have to has the same conversion between the crypt and your plugin, and also the library. Dependency must be the same and also should have the same goal path for the creeper and your plug-in.

A

It is harder to maintain in the development.

A

So user also will use creeper in another two scenarios. uh Here is one example. The first is lu. Engine rose engine so edge x, foundry in the right side. The creeper projector was uh the referenced row. Engine implementation in the address foundry project.

A

So from the april 2020 it has been included in the address foundry project. So if you try to use the address foundry, the creeper is also included in the project and then the next is the data format and the protocol conversion.

A

So for some of the reasons a user wanted to accept the data from source and then after processing sends the data to another system, but the two systems has different protocols or data type, so user can use the creep to do some transformation and the conversion.

A

So one of the examples is that user user sap netweaver rc sdk to extract the data from the sap system and then send it to another system after processed by the creeper rules.

A

Now in this page I will introduce a little bit more about creep, coupe edge and equipped integration, so the kube edge is an open source project, extending native containerized application, orchestration capabilities to host at edge with quip and kube edge integration. It enhances the edge analytics capabilities so.

A

The benefits for the two open source project integration is that it addressed the iot edge computing challenges, so with creeper capabilities provided for edge analytics, it is, it provides lower latency and the bandwidth causes saving, because all of the data are can be processed as an edge and also.

A

Therefore, it is easy for user to implement or refreshes the business logic, because user can just change the creeper rule at the cloud site and the last benefit comes from the coverage user can manage and deploy the creep applications or ar aggregation from the cloud and also manage the rule from the card.

A

So it address the three challenges from the.

A

For for the iot edge computing.

A

Here is a customer case that leverages a group edge and a creep, so creeper accepts the data collecting from the device connected to the kubernetes and after processing. One rule will save the data to the infra db, which is a time series db deployed as an edge and another rule was deployed for synchronized data or sending the data from the edge to the remote mqtt broke, which is deployed as the cloud side.

A

The next step for the quick project, so we will contribute the creep to the lf edge, now is under the submission process and also we will collaborate with more of open source projects.

A

A

The last example we will introduce more features in this year, for example, the third party language plugin development support, as I mentioned in the previous page, so the native approach of plugin development is not very convenient, so we will support the third party language, plugin development, which will be more user-friendly to the plugin development and also we will persistent.

A

We will add a persistent support with third-party frameworks, such as rallies and for more detailed information. Please refer to 2021 roadmap. You can click as a link for to get more detailed information.

A

Okay, that's all for today's session. If you have any, let's discuss.

A