Cloud Native Computing Foundation CNCF Webinars, 19 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Webinar: Introducing Kubernetes IoT Edge working group

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

With Edge computing becoming an emerging trend, we see more and more developers trying to use Kubernetes outside typical data center deployments. And it makes sense, as people want to continue using all cloud-native best practices and tools that make them productive. IoT Edge working group introduces a community where all aspects of using Kubernetes in IoT and Edge use cases is discussed. In this session, we’ll introduce the working group, share some of the interesting topics discussed so far and provide an invitation to participate in future activities.

A

All right: well, why don't we go ahead and get started? I'd like to thank everybody for joining us today, and this CN CF webinar is introducing kubernetes IOT edge working group. My name is Randi Abernathy I'm, a CNC F ambassador and a managing partner at rxm and I'm gonna moderate the webinar today and I'd like to welcome our presenter day on buzz net software engineer at Red, Hat and Before. We jump in though I just want to hit a few housekeeping items. The attendees can't enable audio during the conference, but there is a QA box.

A

So if you look down at the bottom of the zoom screen, you'll see QA and in the QA you can drop in questions and then at the end, I'll kind of go through the questions and we'll see if we can get them to give us some some answers as much as time as we have for running through the questions at the end, so drop any questions that you've got in there as we go through the material. The session is being recorded also, and so that's going to be sent out.

A

As you can see in the chat there, there's a link in the chat and it'll also go out by email. So if you were signed up for the seminar, you'll go get that as well. Okay! Well with with that out of the way I'll hand it over to Dejan. Thank you.

B

So Kai first, a brief introduction to me, as Wendy said I'm, a senior engineer at Breathitt working on the IOT and messaging team and I've been around for the last I. Don't know how it is like nine months since we started the IOT edge working group within the Kinetoscope community and I would like to present a little bit more more more about that today.

B

So the reason we started this working group is that in the last couple of years we we see that you know more and more people are trying to deploy IOT and edge solutions using cuban entities, and there was a lot of questions being us on how we can use the u.s. kubernetes is a platform for for all these confuse cases, from trying to deploy IOT cloud platforms in in the communities to deploy repoing certain nodes in the area or the whole cluster, so on the edge.

B

So the idea was to form a working group as a community where we can discuss all this topic around around IOT and and edge computing and first you know to start with with you know what whatevs means and and kind of established like a common vocabulary, because people coming from from different industries from different areas when people say edge, didn't they mean different things. So so we started first with the idea to to bring all this concept together and and try to distill, what's relevant in the in the kubernetes concepts.

B

So we tried it to collect the use cases, the workloads and applications that are better common to to the communities on the edge. Try to you know answer the most important question is and that's. Why would you use communities in such an environment? You know, but is it a good fit? Is the technology at all or you know, is there a better technology solutions that could be used used and then you know. Finally, when we start looking at how we're gonna use communities to run those workloads in the IOT?

B

In that context, what are the key challenges? People are facing to do it because once you, you you're, moved out you're, you know centralized data center. There are different aspects that that that will affect how our cabinet is work in these environments. So, oh, that's.

B

What I want to discuss a little bit today, just to you know, share some things where we are with these discussions and and what are our plans for the future, and hopefully I can get you interested in two topics and to join us in this working group to shape this this area more in the future. So we'll start with a brief introduction to the edge computing in kubernetes. We won't go too deep into this, because there's a lot of other places where, where people are out, you know defining the edge computing in general.

B

So, but we need to do it. Do this in order to, as I said, define the common vocabulary and to understand instead, what we mean when we mentioned certain terms, then we'll go through a typical workloads that we intend to fight that are useful and people trying to run things on the edge with kubernetes discuss.

B

What are the challenges when you try to use kubernetes in this context, and then, finally, you know shout out to a couple of projects that that are starting to appear to address some of these issues, issues in the in the landscape and finally talk a little bit about where what this particular group will focus in the into coming months.

B

So what what's that edge computing gets the first question, and one like poking leigh-anne stories like edge, is everything that's not located in the cloud right, but to be more precise, we can define edge as an effort to deploy workloads, services and applications closer to the users and devices that are basically originators or the data for those services and ultimate consumers of the data use provide by those those services.

B

Why would you want to do that? That's and other valid questions, so why? Why would you even consider deploying something outside of the centralized centers club, so there's a couple of aspects that usually pops up when mental conversations starting on this topic, one of the most prominent is tackling the latency. So all the late latency sensitive workloads are better deployed as close as possible to the users and devices that needs those services.

B

Other is availability so deploying deploying the world clothes on the edge side and on the sides that doesn't have bulletproof connectivity to the to the central cloud, but making them work properly. If even if they're offline in serving the the users and devices in those sites is one of the crucial use case, data locality.

B

More and more, we see the policies that that make moving and sharing large amounts of data popping up up. So in some cases you might want your data to never leave your factory or your enterprise location, your your remote sites, so you want to have a processing of the data locally, but you still want to use our cloud native development styles. So in that case you might want to consider deploying thoughts and workloads on the edge sites and the list goes on.

B

But these these are basically the three main three main: the incentives for people to even start considering doing doing edge computing use cases. You know coming from two previous slides. We can see, but this can fit in into various industries and and existing industries like IOT, for example, there're always.

B

Kind of a way in IOT to create gateways that you'll do some data pre-processing and things like that. So you we see the trend that that more and more people want to containerize those those gateways and manage them using the cloud resources. So that's the natural fit. So all the other providers also want to use their infrastructure to provide new services, so telcos ip's CDN providers already have a lot of a lot of adrene's infrastructure so to say so.

B

Why not use that to to serve some of the applications better to the local users and in the terms of latency video game servers are probably the best example to explain: explain the latency, latency requirements for edge applications and the compute offload doing intensive computing for the VR or a AI can be can be offloaded to to the edge sites as as well so.

B

The next typical question is is like everybody's saying, edge, thinking about different things, so I think it's important to distinguish what what edge means to different people- and here usually you can. You can divide things in the two separate categories. First, one is the infrastructure edge I.

B

Also read somewhere today, the term heavy edge which with also could describe this this category pretty well, and you can see this is this- is the edge so edge site which is outside of the public centralized cloud, but but but it still have a good networking infrastructure and and basically more more resources there. So when we talk about this in inter Cuba net is kubernetes context, you can see that that people.

B

The hole called kubernetes clusters on the edge sites talking about hybrid clouds or federated clusters, kind of use cases there. But then we can. We can go even one step further and and deploy our computer resources even closer to the to the users and devices. And- and here we usually talk about like enterprise, retail stores, offices, factories and things like that, and and here we have a different again different networking requirements and different compute resources that are available. And- and here we can talk about two things.

B

So we can also talk about deploying the kubernetes clusters in this areas, but also their use. Cases where you want to share like a pure IOT solution, where your edge nodes are basically just that kubernetes nodes, not not the whole cluster, but having having the cluster nodes, managed by the same control plane as the rest of the rest of the cluster infrastructure.

B

Again, another, why I think it's important to ask a lot of wise when you talk about things like this, and- and this is like- why would you want to want to use cuber Nettie's for all this? So we see that you know what cognitive computing is is where the developers are today and all the tooling and and all the best practices are being developed for that.

B

So so it makes sense to try to reuse all that effort and and and and just to develop and deploy a little bit different kind of work workloads for for the edge right. So what why you we went event or something completely orthogonal to something we already already here today.

B

The other thing is that you brunette this architecture is pretty good for this kind of use cases, even if not no matter what kind of scenario you're talking about deploying the whole cluster, so the edge or or the point, the only certain nodes on the edge a lot of again a lot of tolling. A lot of works had already done on this front, so it makes sense to try to reuse all that to take out all these new deployment problems.

B

Now we can talk a little bit in general about what are the challenges are when you start moving your workloads from the centralized cloud to the to the edge and as far as I can see, you can broadly categorize these challenges into three three main groups, so the first group is is like an infrastructure plane, and this is always the case when you talk about any kind of IOT and edge computing it, and that's like how do you put your computer resources on the edge and how do you install them?

B

So it's a those edge sites are not usually not typical data centers. So there's a lot of considerations to be taking care of of you know how you put your resources there, how you secure them and how? How do you, you, you know, create a proper, proper cluster or proper infrastructure on the edge side.

B

The other one is the control plane. So now that we have our resources and reaching out to the edge, how do we schedule our workload there and how? How do we manage them, because the one of the recurring patterns going on is, is that you know the the.

A

B

Between the agent and the cloud is different, so so all all the things that that we take take taking for granted in in the cloud are are a little bit different now, so we need to. We need to think a bit think a little bit more about it and finally, there's a data plane challenge, and this is how these edge workloads communicate with with a cloud and even how do we enable edge sites to communicate with each other? So.

B

So far we hit a little bit so I'll talk now a little bit on around where we are at the moment around discussing all these things and in how do we define them, and hopefully that will help putting all things more in the perspective. So so, first we'll start by talking about you know what kind of what kind of workouts are suitable and two to be run in this in this context, but before we start with that, that.

B

One of the things that that that needs to be mentioned is that there are a lot of features in Inc unity. Shtetl, you know usually not use that much in the pure cloud environment, because simply they are not that important. Those kind of use cases. So so, when you start talking about about about the edge computing and using kubernetes in that space, some of those features features become become more important and we'll talk about a little bit more about those later on.

B

But one of the first things that that we can see is like how can we? How can we use how we can be influenced so keeping at the scheduler to better better schedule, workloads to the edge nodes? For example, if you have a constraint constraint note that only certain workloads needs to run, then we can. We can use paints and durations to to basically instruct the scheduler tool to put only only appropriate workloads to the to the edge node.

B

So, as you can see, we can taint the node with a cube Castle comment saying that that only only the pods that that that here equals true in this. Basically, examples Toleration on themselves should be considered to be scheduled on this particular nodes. So when we haven't are now defined like this, we can basically see in the above above the two pod specifications and on the left side the specification Java toleration for for for for this, so meaning that that this pod is able to run on the on the edge node.

B

While you know any other node without that, without that toleration, for example, the the pod on the right will never be will never be considered to be scheduled, scheduled on the on the edge node.

B

So this is you know just one of the examples of of, and we will talk about, paints and toleration is a little bit more in these workbooks or procession, so so I thought it would be useful to just you know, share it here up front to give it a bit more context on how that that kind of that can be used to to schedule schedule workloads on the on the edge notes.

B

So if we again take a look at the different kind of workloads and and and try to answer the same old y question, so what you know, why are we deploying workloads on the edge and again there's a three broad categories so far identified? So you know we're closed if you help help us with data ingestion and pre-processing the rules that will help us with reliability and availability of the edge site and, finally, the the workloads that will help us have a better latency for the photo for the applications and workloads that need it.

B

So, let's, let's dig a little bit deeper into this, so so what what I mean by the word holes did that did that help with with the data injection. So in many LT use cases we have a bunch of devices that are talking of what we call like a field protocols and there's a lot of a lot of these protocols. More a lot of them even IP.

B

Based so- and this is the traditional job of the IOT gateway- is to do like IP onboarding of all these kind of devices getting the data from from them and putting putting them into something. That's that's more communing to be consumed by the cloud. Basically right.

B

So a typical example in the industrial IOT would be like a the Modbus all in consumer IOT, the Bluetooth, so so the nodes that would that that would do this kind of job would get a data from the devices and convert those into some more common common protocol like Inka, DT or HTTP or AMQP right, and send those data to be processed process further down the stack.

B

So, as I said, one of the couple of things that the cuban Nettie's provides as features less known to do, this is device plugins, so device plugins allows allows device manufacturers to to basically write plugins to to advertise date, their their their hardware to the coop, cubelet and, and then the the pots and containers for Claus can request those those features, and we can also use the taint scintillations to to basically make sure that that results are our schedule to the appropriate, appropriate nodes that it can that have appropriate features right.

B

So once we have this data.

B

In an appropriate format that we we can use for processing, we can do do some more pre-processing or data on the edge edge as well and there's a there's a two things why you would you want to do it so or even more so we can do a basic normalization of the data structure read by devices to make it easier to to identify the context of the data later on in the applications, and we can add a lot of metadata about location, identity of origin of the of the data at necessary security context to the data, but also for for more advanced use cases when you, as you said earlier, sometimes the bandwidth between between the edge sites and and in the cloud is constrained or very expensive.

B

We can use some data analytics on the edge side site as well, so so to doing some data, pre-processing data analytics and sending only the relevant data up for processing, so basically, basically trying to to two separate noise from from the actual use useful data there. Well, as the slide says, so, there's nothing nothing specific about these workloads again. They need to be properly containerized and and scheduled to to appropriate at edge containers, edge nodes or cluster, depending on the on the particular deployment.

B

Reliability in high availability, so.

B

Just to remind it that we, we said that, as is one of the the main use cases for edge computing, so being able to operate in the offline mode for for up for prolonged periods of time and considering that that a normal thing so for that, you usually want to be able to to store some of the data on the on the edge side. And usually we you know, storing data on temporarily on the side is, is is a to waiting.

B

So so you can basically do Bufferin batch or store-and-forward, meaning that that your devices and applications will continue producing data and sending like everything is working. Fine, even if the it if the connections is down, so that this means that we need to to deploy some kind of the message brokering on the on the edge sides, so that the the broker buffers all the all the all the traffic. And then you know scientists and sends it upload went mental.

B

The link is up right, other is providing the data for the services running running locally, so so doing some kind of caching, caching of the of the out of the state of the applications running in indeed in the applications locally on the on the nodes, so that workloads and services services can continue working working normally, even if, if the site is, is all fine.

B

So one of the the biggest considerations here is, we will talk a little bit about it in in a few slides, but having limited capacity capacity is usually what we have on the edge so so making sure that that all these workloads, having enough storage, for example, to hold enough data for the for the plant offline offline activity, right that that is the main being concern for for this particular use case.

B

Service is, is all the rage these days and it makes sense for for latency to try to to to see how we can. We can deploy functions locally on the other side, so they can. They can react on the sensors sensor or user events or or even on the scheduled event. So that's that's one of the big areas that can be explored in this. In this context, and also for the portal agency, we see that that there are a lot of interesting in running.

B

Machine learning on the edge, so you can see those of the those of two different approaches to that. Basically deploying the cloud train models which will be executed on the edge and the other one is edge, specific training which is which is conditioned on the environment and the data policy for them for the particular site. Again.

B

Here we can, we can still use device, plug-ins and paints and tolerances to make sure that that appropriate workloads are are scheduled to the now it's a bit with with the appropriate capabilities. So.

B

Now that we went through like a basic crash, so why would you? Why would you want to do that? There's always a question always a question of what? What are the challenges, because each time you step out of your centralized route, you get a very different system right. So two main things are usually discussed in this context and that's des the limited computer resources on the edge side and the problem with with with the network, so network ambient and secure and reliable and unlimited him in bed in bed with and in bandwidth.

B

So in the next couple of slides, I'll go through again couple of humanities features that are less mentioned in in the in the pure cloud space, but could be a helpful tool to mitigate these challenges on the edge. So.

B

When you're running in the cloud, auto-scaling is something that that is usually take taking for granted. So if you need more resources, it's usually fairly easy to add, more nodes and and and the scale scale out. Your your solution but, as we said, deploying things on the edge is, is more complicated, so so, basically you're you're ending up with bit having the limited number of nodes there. So that's where we can use a prior occasion, prioritization talk it you.

B

It could help us to make sure that important workloads are always scheduled, proper but properly on the edge notes and there's there's a 2-2 aspect to this. So so we can. We can define priority classes for our workloads and basically define them as as a assignment to a different different pots, and when, when, when the pot has, he has a priority and when it goes to the scheduler to be scheduled, scheduled on a search on a certain note.

B

If, if there is no, if it can be scheduled, sound some of the some of the less priority workloads, it will be evicted will be preempted preempted for fall.

B

From that note, in order to do allows us to to schedule more priority workloads, the other, the other feature is called polecat service, and this is not the explicit but but rather implicit configuration for the workloads, so each each word each container in the pot can help like a very set CPU and memory limits, and if both of those limits are set on all the containers, we can say say that the quality of service, for this particular part is is said to be guaranteed.

B

If one is some of the containers, you have the limits, then the poles are are considered considered. Worst of all, and in the fault case, if we don't have any any limits, any requirements set for the containers and then the workload is is considered to be the best effort. So with these two, these two with these two, the properties of the pots we can basically.

B

Configure our system to to to to have like a properly properly important and high priority pots being being considered as a as a as a first class citizen by the by the scheduler, and then you know some some some less priority workloads like, like you know some things that are that should be running in the indy off hours and and maintenance. Those parts and things like that can be, can be considered considered with the last less priority.

B

The other thing that's always important is: is the networking between the between the edge sites and end nodes. This isn't so much a concern for a-- for a fool. Let's say: infrastructure edge where things are running in in the service providers, but even for the enterprise edge. Where, where you can you want your rules to run in a retail store, those things usually have isometrics natural capabilities and all the workloads running there are basically competing for the same same resource right.

B

So here again, we want to hear some some kind of tools to like limit the bandwidth to a certain workloads and give the priority for depended for the for the for the. For the more priority priority work workloads so first, all we have here is is a natural policy which base define, define a little bit and help us a little bit with the security of the edge workloads, because it can be used to apply a certain like like a firewall rules to two ports selected by the by the port selector.

B

So we can define the rules and in the set of IP addresses that can be, they can be used for ingress and egress and address for the certain certain certain certain ports which is, which is an important issue when you, when you're running in into in the in the untrusted edge environment, the traffic shaping the other topic for the network is always the bandwidth.

B

As I said, a lot of workloads are are competing for the same it networking resources and even if the, if the bandwidth camp can be, can be restricted on the number of layers in the system.

B

What we are usually most interested in is how do we constrain a certain node and in the pods with with with with in that node with a certain bandwidth and that's where the steam scene I plug in help us, because it can be in a beset with the with the benefit capability set to true. Basically, CNI allows us to define the the networking interface is that the cubic unit is node, we'll use, and we we we did. We can, as you can see.

B

In this example, we can set basically increase rate and a growth rates for further for the node node itself over the certain interface, which can be useful in a scenario where, where you can maybe have a backup backup interface going over over slow and expensive, like a cellular network or or set satellite net network. So so it makes sense to to try to make sure that that the bandwidth is kept when using the interface.

B

But when the bended capability is turned on, we can then annotate our ports, even in the running, a certain node and basically limit the number of of the bandwidth. Those those pods will be using when, when running on a on a certain on a certain node.

B

So this was just a little bit of examples of you know what what what water? You know, what what are the challenges and what people are thinking about, where they're thinking about scheduling things and deploying them in the edge environments. But, as we said, there's a there's a lot of things going on in this.

B

In the community and there's the interest in the edge computing and lot of projects are popping up, so basically that the whole kubernetes platform is struggling with three main things, and it's it's as I said infrastructure countries how to to deploy its plus process on the edge, how to deal with with unreliable network between the control plane and the nodes, and and how do we deal with with those constrained resources so for the rest of this session?

B

I will just call out couple of these products and try to share what L, what exactly the problems are they trying to solve so in this area. So, first we'll start with with a cubic which is an IOT edge edge platform.

B

It's it tries to tackle the device edge problem, an IOT edge problem so having the central central control plane, managing the the constraint constraint devices with the widow saying kubernetes api, and you know in order to solve that problem, they are providing a lot of a lot of components and relating hypnotists services like like cuba's which which allows the cubic unit is the control plane to work.

B

Even in this kind of environments and enabled enabling talk to schedule the supports on the on the on the lightweight lightweight agents and and controlling them over, the unreliable unreliable network which are qubits is on the first look similar thing, but it's it focuses only on on a single single job and it is to provide a framework for implementing alternative built implementations. So it's it came from a different origin.

B

It came from the use case of trying to scale out the kubernetes cluster to the resources to the other resources in the center, but you can see that which virtual qubit can be used to implement accurate api for device, since that that's, for any reason, can be a proper, proper kubernetes node. In one of the example, one of the example sort of you in vitro cubed.

B

This context is a sure IOT edge connector for for kubernetes. So so you can take a look, look a bit more about it. There k3s is also recent project by venture, and it focuses focuses primarily on solving the infrastructure problems so solving about how to deploy cuban ideas clusters in the in the in the environment. So it provides like light lightweight kubernetes distribute distribution which should be easy to deploy on earth should be used to easily deploy kubernetes cluster owner owners for a small cluster of devices on the edge.

B

Finally, eclipse I of fog is: is another interesting, interesting projects trying to provide a micro services framework for the edge trying to do to implement a lot of powerful tools and concepts so that developers can easily deploy containers containerized verticals on the edge they are active in the working group as well, and so they're moving their their implementation towards the kubernetes kubernetes solution.

B

So that's basically, where we are at the moment with this working group. So all these topics have been discussed through the regular course and and and other resources. The group here and try to be. You know captured in form of this, this presentation, deck deck and in the white paper which we try to maintain- and this is you know, just just beginnings, so this is kind of trying to be like a introductory mud material for everybody, new coming coming to this area and trying to take out these kind of issues in the community context.

B

So in the future, what we're talking about now is digging into a certain certain topics of certain aspects of all this and trying to provide again best bad practices and a blueprint of how we gonna solve particular challenges. So toting is that popped up is the security and the the data paint messaging aspect of of edge computing that it will be discussed in the future.

B

So here are a couple of a couple of links, useful links to the working group, so we meet every two weeks and- and we alternate alternate between two time zones in order to to have a better cover for for both Weston II is Facebook's.

B

Next meeting will be next Wednesday I think then it a m-- pacific.

B

You can always reach us on the message group and and slack, and you know, will be- will be presented at the next cube con. You trying to prepare it more more material by then and talk to more people in in in person, so I hope I got you a bit interested in into into the topic. So so, if you have any questions pop up contest like an and out there there's a lot of folks ready for any kind of discussions.

B

You have any interesting use case, that's also very valuable, so we would like to hear from you and that's all I had for today. Thank you, hey.

A

On thank you so much the great presentation, and also thanks a ton for you know putting all the energy into this this this group, it's really important stuff, and it's it's. It's wonderful to have people working to try to avoid duplicate efforts and things like that in this space, things are moving so fast just want to. Let everybody know call out for questions if you have any questions, just go ahead and click on the Q&A icon down at the bottom of the zoom screen and drop them right in there and we'll we'll go through those.

A

As we have time, we just have probably a pretty small amount of time here at the end. To pick up any questions that come in, but one thing in the in the Q&A there Dan is that people are interested in materials around this stuff. The sli's things like that so I know you have like a white paper linked in here are these? Is this particular deck and I'm pretty sure it's going to come out with the link for the presentation, but is this deck or for the webinar? But is this deck available online as well?

A

B

In another forum, yes, but I will I will share with with a group and and put it so I think the white paper here a lot of our resources section where there's a lot of other white papers and in the materials general available for the edge topic, which I think could be interesting for the people interested into that and basically yeah, just monitoring that the meeting notes and agendas people can find a lot of presentations that that we already had during the the work group meetings and and all the materials our felt share there.

B

But I'll definitely share this with you guys and we can. We can. You know.

A

Fantastic, so it'll come to you since you have channels, but also it'll, be in the in the white paper down in the resources section and I. Imagine it's kind of a living breathing document. Right I mean it's gonna have all this. You know more ideas and more kind of solidity comes to different parts of the equation. Right, yeah.

B

So we started the white paper, basically just as a as a as a means to to come to the as I said, common vocabulary and just a way to understand each other. So later on. The work shifted a little bit toward this kind of slide presentation format.

B

But we'll try to put the resources back into the white paper too, and publish it and I think it should be a important deliverable to have this kind of concise introduction to the topic somewhere and and have it like with the with the with a constraint topic and then, as I said, all other topics that we're trying to tackle next you'll be will be presented it at a different kind of white papers or demos or blueprints. You know whatever group comes up with.

A

Gotcha and you know, a lot of a lot of folks are sort of chiming in on the vagueness of the overall edge architecture, and you and I were kind of talking before the the webinar a little bit about this there's. You know 50 million different solutions out there and they all kind of have different ways of doing things. Do you do you see that ever coalescing or to some degree? What are your thoughts on? You know, kind of archetypal. You know sort of architectures yeah.

B

I I think it's it's hard to say because, as you can see from this it really what edge means to different people is kind of completely different kind of things, so I think.

B

In the end, there will be a couple of different, no different architectures provided for different kind of use cases, so we can see like a qubit taking only the IOT use case, but other people are interested in the edge in the retail. So it will be hard to come up with with a single single single, so to say, platform to solve all the particular problems, but I think having all these kind of tools will help people to actually you know solve this use cases depending on on their needs. Yeah.

A

And you guys have started the discussion, which is great. You know so that all of the the the lint can come out of the equation and and get sifted. There's a question here that that ties into that so for specifically for telco folks, who obviously care a lot about networking. Are you guys looking at tackling some of those challenges and trying to provide guidance around around that aspect? So.

B

You know the group is up open for everybody that hasn't a topic that that one wants to discuss. Of course it's it's on the agenda. We haven't tackled tackled it yet, but you know it's is there catch it? So if anybody wants to join and start a conversation around it, that will be awesome right. What's.

A

The best way for people to join just show up yeah.

B

Just show up so the job is completely open. There's there's we try to keep the product marketing at a minimum, so try to keep the discussions as technical as possible. So it's best to join the flag and to join the meetings, and you know start from there. Gotcha.

A

And this may be time for one more question: do you see the the propagation of lightweight container kinds of technologies for the edge being important? Yes,.

B

Yeah III I think in the certain use cases which we already see that, for example, you know looking at the IOT gate piece. You know people just want to use container containers and, and the containers were closed, so I think we will see more and more efforts being put into that there. You guys know gotcha.

A

Great well Dan. Thank you very much and folks I think we got them. Most of the questions may be missed, missed one of them, so apologize. We didn't have time for all of them, but feel free to you know, engage through you know all the resources that are provided here and and again thanks everybody for attending. Thank.

B

You for having me- and you know, for all the questions yet the best thing is to ask directly under on the working group links, and there would be a lot of folks there willing to discuss all the aspects of this. Thanks again for having me all right, bye-bye, everybody.