Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Energizing the Manufacturing Industry With Kubernetes And Cloud Native - Marcel Wagner, Intel

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Energizing the Manufacturing Industry With Kubernetes And Cloud Native - Marcel Wagner, Intel

Speakers: Marcel Wagner
We present the Cloud Native Digital Twin architecture of IndustryFusion Foundation (IFF). The goal of IFF is to develop an open-source platform to digitize factory processes and achieve carbon neutrality in metal processing. From the start, IFF chose Cloud Native and Kubernetes as the foundation of all workload management. The machine gateways, the factory servers, and cloud deployments all run on CNCF certified K8s clusters and are, therefore, independent of a specific Cloud or Edge Service Provider. The Digital Twin architecture is fully declarative, based on Semantic Web frameworks like RDF and JSON-LD. We developed a special K8s operator to monitor and deploy the factory processes as Apache Flink jobs, defined with Streaming SQL. We will also describe our experience running K8s and Cloud Native in a manufacturing environment, especially the challenges we overcame with the machine builders and factory owners to make them comfortable with the Cloud Native and K8s approach.

A

Welcome to the presentation, energizing, the manufacturing industry with kubernetes and Cloud native. My name is I'm working, Intel's, Network and Edge group as software architect and I'm located in Munich Germany.

A

The agenda of my presentation is as follows: first I will give context and motivation why Cloud native is relevant for industrial and Manufacturing I know I will provide semantic web standards in a nutshell, because it turns out that semantics becomes more and more important in the space. I will show. The cloud native architecture derived by the institution project gave an overview about how kubernetes operators are used, especially how a new one has been developed to support analytics in the semantic space.

A

I will provide a demo give my conclusion, and after that, hopefully we will have a live q a session.

A

One of my job tasks is to work with small and medium businesses in Europe in the manufacturing and industrial space to support them to adapt new software Technologies smaller meeting businesses for All Purpose are companies with less than 250 employees.

A

But when you look at the overall statistics, you will realize that their contribution to the added value in the economy is huge. Therefore, they are very interesting to work with, and the structure is very special, normally they're family owned they're very focused. Some of them are even market leader. In specific niches in the industrial space, the challenge is that many of them just started their industry photo to Journey.

A

So when you look at their factories, you still see a lot of heterogeneous machines from different vendors, even very old machines, and it is very, very tough to provide environment where all everything works together and also there is not yet a common open platform where everyone feels comfortable, because one fear of the smbs in industrial is to lose control over their data if they provide it to a third party.

A

In order to address that more and more smaller, medium Enterprises in different domains are getting together and creating organizations to define the rules of their digitization.

A

One of them is industry. Fusion Foundation, which allowed me to report about their project state in his abusion foundation, consists of around 60 manufacturers and stakeholders in the metal processing industry in Germany, they're, very focused, metal processing or metal sheet processing. Cutting welding deburring is main things they're doing.

A

Of course, they want to align with the overall industry 4.0 activities in Germany and Europe, but at the same time they want to own the base platform.

A

That means they want to own the semantics of the machines and the semantics of the use cases in order to address a wider space and collaborate with more ecosystem partners, they decided to open source the project and another very interesting decision was to go from cloud native and see what of this. Technology can be applied to the factory.

A

In Munich, you can book a tour in a factory of a very famous car manufacturer and watch how dancing robots highly automated, creating very nice cars, and you might think everything is done right. Automation is a no-brainer. Everything is optimized and there is no big issue with industry 4.0, but then you go 50 kilometers outside of Munich into a smaller Factory and you're more likely to see something like this.

A

So this is showing a very simple cutting metal sheet, cutting use case or setup, so the machines are cutting, but when they are cutting metal sheets, the the safety regulation demands that the air is filtered.

A

So today this regulation is fulfilled by a drive wire, so the filter runs on maximum and when the cutter is on, the filter is on. But apparently this is not very economical and not very highly automated.

A

When you knew more about the whole process like, for instance, what is the dimension of the work piece? um What is the material of the work piece? Then? You could save money by reducing the filter, strength and power by retaining the air and the factory without blowing it out of the factory and by managing the hazard status of the filter coverage to make it cheaper to when you dispose it.

A

But it's so obvious that you can save money. Why is it not done?

A

There are challenges, One Challenge is, which is maybe not very technical, but still technical, that um when you change such systems, you need a safety certification and that might be in some cases, not even possible or very expensive.

A

But when you also talk to domain experts, experts on the cutting machine expert on the filter, um they have no formal way to talk about their use cases and show the benefits of this.

A

Also, when you look at the use cases, you need to combine information from different domains like material building processes, environmental to create the best benefit out of that, and if you only look at the return of invest, of course, it's not very economical to do only one use case and deploy infrastructure for that. So whatever you do, the framework must be more capable to support. All kinds of use cases, otherwise this infrastructure will never be deployed.

A

The solution the project came up with was to go much Beyond metal, Collections and dashboards. They decided that they need semantic web modeling of the environment. So with somebody web standards, you can add relationships and context to your data. You can relate knowledge or ontologies from different domains, such as material with the knowledge you have in your specific domain.

A

You can even use that to formally assess safety requirements from the former description and you can collaborate on use cases independent of a specific platform, because you have a platform independent description.

A

Therefore semantic web standards are very important in this space and I am not giving a quick overview.

A

The key ingredient of semantic web is the resource description framework and it starts very simple. So everything is a subject, a predicate, an object uh subject and object. Apparently nodes and the predicate is the link between it and then you, when you take this triple structure, you can of course, Define much more complicated or RPG complicated graphs in the right side. You see another example where you can combine typing like um you have a type system like you know.

A

Plasma cutters are derived from cutters and the cutter of this un isoplasma gutter, but you can also Define and describe other things like serial numbers or where the machine is located in is that you can Define pretty complex knowledge, trees, but also models of your environment, and you need a way to further structure it and for this Json LED comes very handy, Json, LED, Json linked data and this Json LED. You can still operate on an existing graph, rdf graph, but you can group it. You can, for instance, say in My.

A

Graph I have a work piece: I have a plasma cutter, I, have a filter, I, have a filter cartridge and with that you can even export or import plasma cutters into your system and Link them with other entities like a filter on the right side. You see an example on Json LD. It is made in a way that it's 100 compatible with Json specification.

A

So all the tools which are able to process Json can also work on Json ID, and it provides you some additional information about the semantic web and this information is typically started within it.

A

But now you have a model, you have information, you have knowledge, but what? What do you do with that? There's a very central question and to Define your constraints and how you operate on the graph there's another language called shapes constraint, language shackle.

A

The second you can now say look. This is my graph. You have rapist plasma, cutter, filter, filter cartridge, but the plasma cutter must be connected exactly once with a filter, and the filter must be of a specific class in this case iff filter. And if this does not happen, it's a severe, very severe problem.

A

You see on the right side an example how chuckle looks like and with that language you can of course look for much more complicated relationships in your graph.

A

For instance, you can make a statement about the plasma cutter has a state and the filter as a state. You can search for it and relate it and say when the plasma cutters on the filter must be on.

A

Another example is when you, the plasma cutter, is processing a workpiece. You can derive from the material of the work piece, the hazard or the East glass of the filter cartridge, because everything is linked. You have only to formulate first of all, what states are problematic and what manipulations or what changes you want to do on the graph.

A

To summarize it um The rdf, Source description framework is needed to Define, merge, graphs and context. Json LD is heading to serialize the source description framework in a human readable way and also grouping relevant parts of the model. For you and chuckle is the language where you can Define constraints or knowledge, graphs and entities.

A

You can Define constraints between entities and you can also Define rules how to operate and how to change the graph. So with that, you have a very expressive way to Define use cases and, of course, constraints, for instance, for safety and have a language and a platform to collaborate in a specific domain.

A

Now, when you look at such a system right, you have huge graphs and those graphs are updated by the data coming from the machines.

A

um You see it's looking very closely to a streaming analytics problem, but not any not only need to map the information in real time to the graph.

A

You have also to check every single event coming in whether it's having or creating a problem, whether the whole system is still fulfilling the constraints and if not, what has to be done. The knowledge graph so that domain knowledge might might be huge.

A

So um do you better have a scalable system and then, when you look at chuckle, you will realize that it's a pretty declarative system. So you define what states you want to have, what not and what to what to do. If you are in a specific state.

A

So whatever this system you derive, it must be able to work in the declarative. Space. Low latency is needed, and one rule for Windows diffusion, Foundation was to say when you have two Frameworks select, always the more generic one which has the broader communities, because they want to the technology it stays for longer time.

A

The decision which was made was to use graph.

A

Going towards relational using streaming, SQL scalability stateful data processing, low latency, is fulfilled by using a batch of link and Apache Kafka and um from the beginning, we derived also the requirement to do as much as possible with kubernetes, so that led to a simplified architecture which you can see here.

A

Filter and cutting system are sending data by mqtt 2, which is bridged to Kafka and on Kafka. You have Apache feeling operating on the data which comes in the concrete logic. How effective link operates is provided by an industry Fusion semantic operator I will show how we derived from the semantic descriptions the actionable processing Pipelines.

A

The time series data is stored in Apache in Apache Cassandra and managed with the K8 Sandra um operator to manage the Json LD part. We have the Scorpio broker, which is also connected to Kafka and which needs postgres for saving the state, and then you need a human being or um a more intelligent application which is receiving the alerts, but also able to configure the system like making sure that the right Qatar is connected with the filter and the model and that the model reflects the reality.

A

But you can also have applications operating on the data models.

A

To derive the actionable processing pipeline, the project develop an operator and a framework how to derive from shuttle, which we just saw, the data description and the data manipulation. Here you see, for instance, that data tables are created for alerts and, of course, for the models like cutter filter and whatever you define in this environment.

A

But then you want also to operate on the data you want to detect whether constraints are um broken violated or you want even to trigger an alert and that's what you see on the right side. So when you, for instance, want to create an alert based on the specific observation. The operation, the SQL operation would be something like insert into alerts, and then you condition your checking.

A

To make this Rock in in general environment the centered it around Apache Kafka, so um we operate on tables right. We derive rules on tables, but actually, when you talk about streams, those tables are topics. So therefore, there's a link between tables and topics, for instance, then you have the cutter or the filter or the machine table. You map them to topics. So when you, then, for instance, the broker is changing a cutter.

A

It will also forward it over the bridge to the topic when you want to get an alert or scene alert to insert something into the alerts table, but actually again it's a Kafka, it's an event on a Kafka topic and is that you can even integrate other services with specific topics.

A

In order to manage those streaming, SQL jobs, the project created several cids and on the left side you see the description table description called beam. Sql table, you can Define, you know columns or fields of your table, but you can also link it with Kafka. I can say that's a Kafka topic, and that is how you reach Kafka.

A

On the right side, you see a structure and a customer source which is called beam. Sql statement set. This is needed to manage all the SQL statements you derive from Shaco, so it can be 20 30, hundreds and put it put them into one object and let the link Optimizer create an optimize pipeline out of that.

A

And now I have a show a demo.

A

A

System to show the demo I'm, starting with showing the Shocker file, which, as discussed, provides all the constraints and the operation on the graph. So here's an example you're saying that there must be a class cutter class cutter has, for instance, a state exactly one state.

A

It has one filter exactly one filter um and the filter must be of class IVF filter. And if this requirement this constraint is not met, it is. uh It has a high severity, but you can also Define much simpler constraints like saying a string must have a specific pattern or a specific length.

A

You can, for instance, the material or you can Define Min and Max values like you, know the depth and have and have a specific Min and max value, but you can also describe more Advanced constraints. This is example, where you make a requirement on the knowledge on the on the graph and you're saying, look at the cutter State and the connected filter, State, and if the cut error is running, the filter is not running then have an alert with a very high severity.

A

So now we have the file with all the constraints, but how does it get into a real pipeline for this initial Fusion developed tools and I'm now executing all these tools in the script and in a second, you will see that after some tests, everything is deployed and now you can, for instance, take a look at.

A

So there are cable, described or deployed dependent on the shutter description and statements, as described, for instance, Checker validation. So when you look at Checker validation, then you see that it's a custom resource containing a list of SQL statement, all SQL statements derived from the shuttle file.

A

So this has been deployed as customer Source. It means it will be collected and deployed as kubernetes as link pipelines. You see pipeline has recently 25 seconds deployed here, the Shocker validation pipeline. When you click on it, you see that it's actually a huge pipeline derived from all the chapter descriptions and all the SQL Scripts and um to show the pipeline the um show the alerter tool which is as described earlier, connected to the respective bridge and in other words with respective alerts, table right now.

A

If I create objects, I can either do it directly from the machines over mqtt or Kafka or I can do it over the broker. uh The Json ID broker. That's what I'm going to do, because it's it's more handy so, for instance, I submit an object describing a plasma cutter, so it has an ID. It says to type plasma cutter and provides the fields which have been defined and chuckled like state workpiece has filter and his filter is defining an URL to a specific filter.

A

So let's submit that and as a result, the analysis happens of the shackle, Pipeline and realizing two problems. One is that the filter that is not connected to um well doesn't connect with Filter. So that is a severe with high criticality mistake. Another not so high constraint.

A

Violation is that there is no work, is connected to a cutter now I submit in the same way a filter and submit it to the system and the same described and with strength and card Regis, and so on and I now submit that you see that the alerts are changing, so the all its severe one is deleted.

A

But now you have a new one, because the filter doesn't have a cartridge and now by deploying the other like the work piece and the cartridge, you see that the model is no compliant, but you can also other do other things like, for instance, now I'm, switching on the filter and um as described earlier now it's comparing it with the state of the switching on sorry, the cutter, and now it's comparing it with the state of the filter and it's a critical error filters off.

A

So either you switch on the off the cutter on the filter. Let's switch on the future with this Json LD API, and with that, as you see, that the model is again in um in okay with the constraints and all all constraints of withhood and there are no alerts created and with that you can create arbitrary relationships on our large graph.

A

So I know switching back. It was a presentation.

A

One question to ask is, of course, does kubernetes fit to Industrial small and medium businesses. From my point of view, the answer is yes: kubernetes helps a lot to unify the operations from Cloud to Edge.

A

This is applicable to a large amount of typical Edge operations in a factory. At the same time, it's also fair to say, there's still a lot of Technologies like real time and safety which are not yet measurable. This kubernetes, maybe that is something for future research. People are questioning the beginning. Do I need kubernetes, do you need Cloud native, but then you can ask them if it's not kubernetes and you want to manage from Cloud to Edge. What else is it and do you think it exists in five years?

A

um The on cloud native? You can also ask them to do the math. What is their aspiration? You start, of course, small, but where do you want to go and how many use cases you need to be uh really have to see a good return of invest, and then you realize that it is a system which must be scalable.

A

Problems exist with small, small and medium businesses. uh They have no idea department and challenges with I.T and therefore also with kubernetes, at the same time, that can be solved by support from organizations like industrial Fusion, Foundation um and also the pure amount of kubernetes tutorials examples, courses and so on is creating a lot of trust.

A

Finally, the line between operation Technologies in the factory and the it part is anyway blurring more and more because the new apps, the new college graduates have touched kubernetes and containers, also at the University at the college and therefore uh are already familiar with that and not afraid of it.

A

The conclusion has shown that semantic web standards and Industrial help to make people collaborate in specific domains, but also across domains helps to assess or pre-assess safety issues, validate data operate on data and Link data from different domains. Together, the industry, Fusion Foundation digital process twin, is using shackle to derive constraints and operation rules, Cloud native hips to have the right, scalability and high availability in operations.

A

Kubernetes operators are used where possible and feasible progress with salando Cassandra with KH sander Apache Kafka stream C about shifting comes with some building kubernetes capabilities, keyclock and mean I will also come with operators, and then another operator had to be developed to manage the translation from semantic descriptions to um a real-time Pipeline on link and Kafka future work. Improved, tooling semantic web is not yet everywhere.

A

um This has to be become more native.

A

um We have to look not only I guess many people look at that how those static or whether they are static, kubernetes, setups, more suitable for real time and safety, and then another Hot Topic is look at semantic web standards to create order to speed up certificate safety certifications. So there are certifiers in the institution project we're looking at such topics in general. Please check it out. Take a look, give us feedback contribute and with that. Thank you very much for your attention.