Kubernetes IoT Edge Working Group, 24 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG IoT Edge 20210324

Description

March 24, 2021 meeting of the Kubernetes IoT Edge Working Group. Open forum discussion followed by a presentation on LFEdge Project EVE by Jason Shepherd. EVE is a universal, open Linux-based operating system for distributed edge computing.

A

Hi welcome this is uh the march 24 meeting of the kubernetes iot edge working group, uh because this group operates under the kubernetes project. We abide by the kubernetes code of conduct. uh The summary of that code of conduct is just be nice to each other.

A

Also under the kubernetes projects. All meetings are public and recorded, so um this meeting will be published on youtube. uh If you don't want to be part of that, recording your options are to drop down or to turn off your microphone camera and change the name in the attendees list.

A

I'm going to put a note to the meeting agenda and notes document, so I'd invite everyone to add their list, their name to the attendees list, and you could help out if you, like, all members, have edit rights to that document.

A

Anyone is welcome to help keep notes. So for today's meeting, uh we've got a guest speaker coming in, but he had a conflict at the first half of this meeting. So he's going to be joining us about 30 minutes in to speak about the lf edge project, elf edge foundations, project eve before we get to that we'll fill time with just free form, built birds of a feather discussion of whatever members want to talk about.

A

I'm going to commandeer the first part of what members want to talk about and bring up the fact that kubecon europe is coming up soon in may and as part of cubecom europe. There is a pre-event called kubernetes on edge um dion, and I have been working on the program committee for that and the schedule is now published on a day full of sessions dealing with kubernetes and edge. So I'd invite you to take a a look at that there is a link to it in the agenda notes document dion.

A

Do you have anything to add.

B

No not much so yeah. Please take a look at the schedule and yeah join the event.

A

Okay, uh if anyone wants to bring up a topic or a question request for help just jump in there and speak also, if there's anybody who's new here, who has never been to a meeting and wants to simply introduce yourself go for it.

C

Hi I'll go, my name is aaron williams, I'm the dev advocate for lf edge. um So- and I will say the the other gentlemen who have joined um are also um you know from project eve, which is one of the projects underneath the umbrella.

C

um I'm really here, uh I'm here just to kind of learn more about what you guys are doing and to kind of pull that information into the wider elf edge community and make sure that we can communicate and don't do duplicate things and things like that. So.

A

Okay, since you're new- maybe we should just go all around and introduce everybody, so I'm steve wong, I'm a software engineer with vmware and I've been a tech lead in this group. For over two years the group is charged with dealing with helping users apply kubernetes for edge use cases.

A

So this group doesn't own code in the kubernetes project, but um we're in a position to listen to user requests and put on materials that would help kubernetes uh up be applied for edge. The scope of that has been traditionally that there are some people who have an edge use case that is pretty full in compute resource such that they can run a kubernetes cluster at edge.

A

There are others kind of in a transitional phase where they might run a split kubernetes cluster that has worker nodes at edge. Maybe a control plane a little more centralized, uh perhaps in a regional center, perhaps in a public cloud and then finally, there is a form that people are applying of utilizing the extensible nature of kubernetes to write crds that would manage to run edge devices that are not kubernetes cluster nodes at all, and I don't think there had.

A

I don't think it's emerged that there is any one right answer out of those three flavors. There are people doing all of them and because these use cases vary quite a bit in what they're trying to do and the resource available.

A

I personally don't know that I'd expect that those ever consolidate it's just legit, to have those three forms of kubernetes. So hopefully that fills you in a little bit about what this group is about. Dion, do you want to go and say a few words.

B

Yeah I mean same here, I I'm an engineer with red hat and being involved into bootstrapping, this working group and and for the last couple of years and yeah. We are basically here trying to to tackle all the use cases where we can use kubernetes for the iot and and uh edge use casing.

B

That story evolved over the over the last couple of years, and it would be great to see you know what you guys from lfe are doing to you know and how we can we can work together. Basically,.

A

Maybe if somebody from lf edge wants to be a spokesperson and describe to us kind of the scope and mission of lf edge and how it might relate to kubernetes.

D

I think this is perfect timing for jason to determine jason.

E

Yeah yeah, I'm sorry I was muted yeah. So let me talk about that real quick. So so um do you? How much do you guys know about lf edge like how 101 should I should I go.

A

Well, let's make it very because not it's not just us, we have people who are going to perhaps catch this on video on youtube later yeah.

E

So I don't think anybody.

A

Who already knows it would be offended by hearing it again so yeah.

E

A

E

Let me pull up a different slide for that's, got more of a high level overview.

A

E

Make you a co-host, so you.

A

Can put that slide up.

E

A

Okay, you got it: okay,.

E

Hold on one second.

E

Okay, here we go so you know, obviously it's a sister umbrella organization to cncf focused on edge. Let me pull this up.

E

F

We can see.

E

That and see my yep cool, so yeah focus on edge. um You know very similar to uh very similar to uh cncf um there. It's focused on kind of the continuum of edge and that's something I think we should kind of talk about when you guys you reference edge.

E

You know what's the scope because, as you guys know, there's there's not one edge, um and so we'll come back to that, but uh all about driving uh consolidation, around kind of common frameworks for edge computing, uh covering the spectrum series of projects you know again very similar to cncf.

E

There is some overlap between projects, but at the same time the goal is to start harmonizing things. uh You know in the future and there's also some unique tradeoffs, depending on what your, uh what you're doing um and so across the continuum of edge, there's just inherent trade-offs.

E

Technically, that's how we define the taxonomy, so I don't know if you guys have seen that but I'll give you the gist of it um every if you, if you go further left from the cloud everything gets more complex hardware, software skill sets, you know everything closer you get to the physical world, so kind of high level project summary so crano is is a little bit different than the other ones.

E

This is about creating solution, blueprints kind of initially a telco focus, but that's moving left a bit they've been collaborating on some blueprints with cncf, as you guys might be aware, but definitely a little different. Some api definitions, some code, but mostly more about those blueprints.

E

We're going to talk today about uh evo s, which is part of project eve and we've been bridging to kubernetes and we're going to talk about the trend there. um There's other projects because open horizon is a container deployment mechanism, but you know probably more complementary to eos than not because you know this is a agent-based solution. Evos is a bottoms-up bare metal solution. So we'll talk about that uh different each project is at different stages, so betels from contributed by baidu is a stage one. You know that's kind of more around data.

E

A little bit more around data analytics and and brings data into the cloud home edge is uh focused on sort of developing home appliance servers for um appliance-based servers for server-based appliances for home use cases. You know, surveillance and natural language, processing and whatnot, and then there's two iot data stacks that you might be familiar with. I actually helped get edgex foundry started launched in 2017 and that's been growing, but then fledge is another application framework for iot workloads.

E

Fledge is more optimized for industrial edgex can do industrial, but it's it's more general purpose. There is overlap at the same time. You know fledge is you know, there's trade-offs. You know, fledge made the trade-off for more compact, higher performance, but less modularity edgex is super modular and so at the expense of you know, light less performance but good for soft real time. So uh working on kind of consolidating and- and you know, creating alignments. You know it takes time, as you guys know, in open source. You know the cream rises to the top.

E

uh There's a couple. Other projects not represented here that have come in at least one actually intel's secure device, onboarding project that wasn't you know here but and I'll share the uh this with you guys. Just so you have it. I mean there's, obviously a lot of stuff online as well. But um so that's that's the mission um you know stages stage one is, is I think this was borrowed.

E

I don't know if this is from cnf or another project within linux foundation, but stage one is at large, um you don't you get the hosting of it stage. Two is where you start getting uh some marketing benefits stage three is is where you know these are are well on their way and you get the most benefits in terms of funding and what not to grow. You further grow the project, um normal technical advisory committee, all the kind of normal setup. You would see an umbrella project, okay, sense in terms of structure.

A

Yeah is the intel project officially in now.

E

Yeah yeah, they.

A

Presented to our group uh a couple months ago, so that's interesting to know also.

E

Yes, it's not on that chart, but yeah it's in here as a stage one.

A

Also, just in terms of scope, you know, like our group scope is that it has to involve kubernetes. You know we're yeah, but other than that, we're pretty broad and open to all flavors of edge use cases. You know whether it be industrial iot or what might be called retail edge, we're kind of not attempting to restrict the scope and stay general, but in lf edge. The words lf, I assume, are attributable to linux foundation.

A

Are you restricting your scope to things at edge running linux, or is it open to other os as well.

E

It would cover, you know, covers everything and that's and that's you know why we think you know evo s as a foundation uh even for kubernetes. It just makes a lot of sense. We would you know your security white paper that you guys put out it literally, like you know that all the things you guys talk about, which is totally accurate, that's what evo west is architected for you know and we'll walk through those benefits. You know eric and roman. You can get deep into the technology, but um real, quick, so level setting.

E

uh Obviously, there's centralized data centers, you know, there's evolution of tools, but pretty well established on the far other extreme you've got super constrained devices that are highly embedded a lot of people from the I.t world say: oh all, compute's happening.

E

You know further upstream and kind of creeping down, but you know we're seeing tiny ml in lightweight devices more fixed function, but but you know uh the way, this white paper, if you haven't seen the yellow fed white paper that we put out last summer, it's a pretty good, read um and it it goes through. You know, there's there's near edge and forage and people say thick and thin or that you know they say industrial edge, retail, there's one continuum and you apply different tools and in in considerations, domain knowledge and that's it.

E

uh The the categories here are based on three uh inflection points. One is: is it on a wide area network or a local area network relative to the users or devices it serves that compute? If you're on a? If you have a latency critical workload, you will never ever put it on wide area. I don't care how fast your 5g stuff is. You're not going to do your airbag from the cloud not going to happen.

E

um The next one is: are you in a physically secure data center, or are you not so that's where this your on-prem data center is your traditional on-prem could be micro, modular data centers once you get out of this you're now accessible, very different considerations.

E

The last trade-off that the paper goes through is: are you capable of abstracting hardware complexity with vms or containers, or is it so constrained, usually driven by memory that you have to go embed it? Everything gets way more complex at that point.

E

Those three inflection points, as is how we define the continuum and the the the way that we're looking at it is if you kind of look at the base um eve, so you guys you got like linux foundation projects, of course, there's more stuff out there, but in terms of kind of example, projects you know open source projects. Zephyr is focused on constrained device edge. You know it's, it's embedded, you know kind of our cause. You have to embed per that uh evo s.

E

The strategy um in that community was start with the lowest common denominator. Go to the hard stop on the left, driven by memory before you just can't support. You know that abstraction anymore, it's a technical trade-off and then evo s, uh support legacy workloads. Do the zero trust security we'll talk about all those benefits, start at that lowest common denominator, and then you're, gonna, you're gonna, see bleed up to to these higher. You know, thicker edges, so to speak, for service provider edge or on-prem data centers.

E

Of course, you guys know that kubernetes, both ka and k3, are moving left, but they start to kind of you run into challenges. Once you start getting into that scale. Factor you get into the outside of the data center or at the fringes of the data center, all the stuff that you guys have been kind of documenting as a team for a while, um and so what we're doing with evos is that we're intersecting the kubernetes trend and because evos supports both virtual machines and containers.

E

You know again we'll go through the architecture, but it's got an embedded hypervisor in it. We can drop kubernetes runtimes directly on evo s and and you're done, but meanwhile you can support native docker containers. You can also support there's a lot of legacy out there. As you guys know you, people are like.

E

I need to deploy a windows existing app right next to whatever, and so it's start with that lowest common denominator security foundation and then meet the data center tools as they come down and we're right at that phase, we're actually about to launch.

E

You know that that kind of formal support for kubernetes, which is why we wanted to you, know meet with the team um you hear so does that make sense kind of in terms of the the continuum and what we're seeing uh in terms of that convergence between like that base and the I.t tools coming down, yeah that does yeah, okay, yeah cool, so so all kind of high-level benefits. uh I've already talked about some yeah.

A

Maybe we can wait because it I told people in our slack channel the eve presentation would start at 30 minutes just in case.

F

A

Postponed and but.

F

Yeah we can cover lf edge.

A

In general, and if anybody else.

F

E

Any questions to.

A

Jump in with right now on lf edge.

E

Yeah, it might be good too yeah, so questions on lfedge. It might be good too, to you know for us to hear a little bit about um kind of the the work output that you guys have had. Since you formed we've seen the security paper, I've seen a number of the presentations you guys have done online, but a little more info would help and yeah. I know we can get. You know really get into it when um people join in, but yeah questions on lf edge, we'll start there.

E

Yeah so clearly, there's there's not one pattern: you're not going to see everything go through the continuum you'll. This is also from the white paper. You'll see you know, device to cloud and gateways uh stuff running at the service provider edge. The way we defined it, those two big buckets, um the user edge tends to be owned and operated by end users uh could be devices that are owned. uh The smart device edge. We, we purposely chose smart device because what number one intelligent edge is is branded by some vendors. But this includes mobile devices.

E

You know pcs, new client stuff, and you know headless things for iot, so gateways hubs, routers servers, the mobile side and client. Pcs. That's a solved problem. You know windows, android ios, whatever the the rest of it wild west. You know it's very, very fragmented and the point with evos is to create that universal base that meets all of the needs for that and then expands up um so so other yeah other questions on lfe.

E

Let me see if there's any other, you know kind of relevant stuff um growing as a community, so growing in members and and deployments um edgex. Actually just hit, um I think, seven million downloads by now. So it's it's definitely growing um a bunch of members and all that uh so other questions anything else. We can kind of cover as we wait for the people to join in eric and roman. You got any kind of background that you'd like to share.

D

No, I think, I think, you've, you know, I think, you've sort of nailed. It.

F

D

No, I I was just going to say I mean I guess my only question is you know, since I'm not familiar with this working group. Sort of you know. Progress so far would be interesting to know like if there's anything that you know is sort of coming coming down the pike so to speak. You know near end guys.

A

Well, let me just go over the history of this group. I think in the early days we started out with a lot of inquiries and presentations on dealing with running kubernetes in a resource-challenged environment. You know, if you, if you're in a large data center with rackmount computers, kubernetes was made for that and it sort of just works with default settings.

A

But as you move down to lower resource you've kind of got to go and tweak a lot of configuration both on your uh worker cluster nodes, as well as the pods themselves to have them declare uh what the resource demands are and to enable the kubernetes scheduler to do triage when uh resources come up short, so that the higher priority ones really do keep running and the lesser priority ones are the ones that take the bulk of the suffering.

A

If you don't have enough resource to go around, then the group seemed to morph into collateral issues like the unique challenges of network connectivity at edge early days. We were home to some of the discussions of things like the cube edge project, but they eventually got accepted into the cncf as their own independent entities.

A

We also hosted presentations on some of the uh lightweight kubernetes distributions, microcase and k3s, but once again I think they established enough of a base that they they are freestanding on their own to a large extent. But this group, I'd like to think, was a contributing element to those things in the formative stages by helping them build community and awareness.

A

So this group is kind of a generalist attitude of helping users uh apply kubernetes uh and you know not focused on any one thing by design we're trying to cover kind of the broadest possible scope, limited only by by the fact that kubernetes should be involved in some way, because we, we are part of the kubernetes project um and I think uh in the last year.

A

The the new trend is this approach of using kubernetes custom resource definitions. You know an ability to plug into the kubernetes control, plane and api to extend it to use it as a control plane for things in general. uh That has really been gaining traction, and this is what allows it to be potentially viable as a control plane for just bare devices of any type, really by any type I mean in any os any scope from the tiniest embedded systems to uh you know full container runtime host uh dion.

A

Do you want to add anything there too.

B

Yes- and I I think you you covered the history pretty well so so what we're also be doing lately is, at least from my perspective, focusing also more on on the on the wider cnc f landscape for doing iot and edge computing. So we we dealt a lot with the k, native, eventing and and so solving kind of uh how to fit, though those frameworks for for for building applications for for for the edge and and iot. So so that was that that was the topic for for the last.

B

I think two two sessions that that that we did for for a coupon right.

A

Yeah, I think people are becoming aware of kind of the challenges of using a traditional data center approach, like tcp connections with tls, maybe not being well suited, perhaps not well suited at all for time. Series data- and uh you know low latency and a number of issues like that.

A

uh Even traditional ethernet transport- is something that, in some of these edge use cases, just isn't it appropriate fit so that supporting things like event, driven message, queuing protocols and things like that- are something that users are very much interested in and there's the potential to use kubernetes uh as a supervisory control plane in those kind of environments kind of if you viewed it as what happened in the data center of something like service mesh being an easy button for not requiring every one of your developers to be a network expert and a security expert.

A

At the same time, if there was a way to put that together to enable secure, reliable communications for edge devices, I think the world is crying out for such a standardization effort. It isn't coming together at this point, but uh we've had a lot of discussion about it. In this group.

F

A

F

A

F

Trying to target the the the or address the issue that you know the network is might be intermittent out at the edge the connectivity back to the cloud and the other components yeah.

A

It would help tolerate internet intermittent networking, and some of these networks are intermittent by design. You know it isn't necessarily reliability, but maybe power consumption, and it is a way to deal with this but, more importantly, I'd say even deal with asynchronous uh uh connections between applications and services such that they don't require.

A

uh You know permanent loan. Some of these don't require low, latency uh connections up to the higher levels they just require.

A

You know eventual delivery of these things and the cncf is hosting a number of projects uh that deal with the this eventing. Like k-native, eventing cloud events, I don't know there's that there's actually a whole category that has uh come under cncf fostership and it plays nicely with these edge use cases. So a lot of this has been, I would contend, an integration effort, okay,.

F

Because we've sort of been looking at that more from the infrastructure side in e right, which is okay, we need to deal with eventual consistency when it comes to, I don't know applying patches to limits right, for instance, right you want to make sure that they get completed reliably and securely. But if this thing is, you know not reachable this hour or this week. Well, it will still happen right right and but but the application layer yeah if people are just used to, I can always stay connected well.

F

Those applications are not going to work across the whole edge right because it depends on connectivity.

A

Yeah there are some of these connectivities. I I don't know like laura that might wake up. In a you know, a microamp uh current consumption embedded device by design. They may wake up once a day or once an hour, and you opportunistically take advantage of your connection. While you can and eventing tends to play nicer with that than trying to run a tls connection.

F

So so uh one question about the security sort of issue is white paper. I've read that thing right is that something that you guys are pursuing any further or sort of? Are you guys trying to make the larger kubernetes community aware of those issues out of the edge, or you know what's happening in that domain?.

A

Well, I think at one time it was a very active thing we published and maybe had asked for an aspirational goal of keeping it as a living document. But, to be honest with you, it took quite a bit of work to bring that thing together and we haven't gone and revisited it. Yet I thank you for the reminder that we probably after having published it.

A

We should schedule perhaps a once every year or once every six months revisit because, like anything in technology, it should be published with a sell-by date and, um to be honest, I haven't taken a look at it lately, but I'm suspecting that some parts of that have become stale.

F

Well, it's also sort of you know it's a checklist saying: okay, where are we at right? Are we making progress on these pieces? And you know, I think that some of the things that we've done in and around eve is addressing the lower layers of the stack right sort of secure boot remote at the station, whatever type of things right and networking pieces as well, but then sort of the applications other things that how do I know that this is actually the workload I'm supposed to be running right.

E

F

um Right, you know, I don't know if there's work in the kubernetes community around around that level of validation.

A

Yeah I kubernetes has gotten so big that I can't contend that I know where everything.

F

A

But uh there is validation of kind of the kubernetes compliance itself, which would be the base kubernetes, but the bigger picture of validating I don't know container images would would have fallen off into image registries and they most definitely do have a policing of the providence of the image how it was built who got their hands on it, as well as looking up the components and mapping them into cves.

A

But that kind of that stuff, uh at least from my perspective, is spread all over in a number of homes. There isn't some centralized authority that has taken control over it um and I'm not saying that it's bad that that doesn't exist because there'd be a trade-off but yeah. I can't contend that. I know that there's one place to go with the broad scope of kubernetes. That would tell you everything you need to know about how you do security.

E

Yeah, let's, let's dive into that, you know I. I definitely could see that, let's, let's dive into just kind of the overview view and keep it conversational, um you know for those of you who weren't um hadn't joined. Yet uh you know I know, there's a recording but yeah.

A

So go ahead and take over in the project.

E

Right yeah, so we're gonna, we're gonna talk about eos and how, as a community, we've architected it to go to the lowest possible common denominator for for edge hardware before you just have to go embed it and- and you know a lot of the stuff happening in cncf you're, great great stuff, but it's presupposing you're you're in a secure environment and the controller is talking with the the um clusters and the controller reaches out to the clusters kind of more traditional data center approach, whereas, as we were saying around the need for autonomy, eos assumes you're going to lose connection and the box always phones home to the controller, and it's got an eventual consistency model which we'll talk about.

E

um So you know it's designed to be universal, including supporting legacy. You meet the needs from a security standpoint. You know, many of which you guys have uncovered in your work. um Lots of legacy out there uh that need for autonomous the the fact that it's bare metal, you can't break devices, there's a number of agent-based solutions. You know both open source and commercial for iot.

E

um You know edge computing that uh you don't unless you spend a lot of time, trying to integrate with the os you're going to brick the device at some point um and then they're all container, based in many regards and a lot of things, including kubernetes benefits being deployed in the vm on top of that bare metal foundation, uh completely open apis.

E

Think of eos for edge as what android did for mobile, um create a open foundation completely open apis, abstract hardware complexity and then enable an ecosystem of hardware providers and app providers to build around it. We've got evos being adopted right now from a lot of hardware providers. You know hpe dell, lenovo advantech, super micro, lanner and app providers, love that because they don't have to touch it uh exactly the same model as android from a standpoint of unifying, but that device makers on the south side and app providers on the north side.

E

So in terms of architecture um yeah, I can let uh you know eric and roman can run through this quickly, but but you know very much curated for these types of edge nodes in a variety of ways. um You know erica and or roman- I don't know if you guys want to kind of dive in on some of the keys. I know this is just a high level, but.

D

Yeah, no absolutely I mean uh I just you know. Maybe we'll uh highlight a few things um and let eric you know talk in more details, but I guess you know the way to think about eos. You know specifically in the kubernetes context right. You know it's all of the missing stuff that people somehow assume you know is just there. You know uh that basically, it needs to be there for uh kubernetes to run and we've seen a number of these operating systems right, you know, ranging from core os.

D

I mean the whole mission of coreos was to basically be the missing. You know link between the kubernetes and the hardware right. You know, then there is talos. There is a few others I mean. Obviously there is a spiritual successor to coreos, which is you know the flat car, something there's clear need for that type of an operating system. Right uh and that's, you know one view that one can take on eve. The other view is actually much more in line with the original mission of um you know.

D

Korres, which is you know, kubernetes, might not actually be the only workload right and uh to jason's point. You know for us, it mainly means legacy right. You know just running an occasional windows application. You know or something like that, um but at the end of the day, it's all the missing stuff that otherwise, as a customer, you would be asking uh your system integration company to do for you, because none of it is available out of the box right, it's kind of like you know this android analogy right.

D

You know it's like yeah I mean you can probably deploy. uh You know red hat or suzy on your uh samsung. You know phone right, but that wouldn't work because well I mean that wouldn't work out of the box because he would still with device drivers and this, and that and like you know, a skilled system. Integration you know, can address all of it, but then you're basically doing this. You know completely undifferentiated heavy lifting every single time you're trying to deploy just a single app. So eve is trying to sort of fill that gap.

D

You know very neatly: it's a very small operating system. You know, unlike some of the alternatives like you know, traditional linux distribution like red hat or you know, ubuntu or you know suzy, uh so the entire eve you know takes 250 megabytes uh worth of storage right. You know for a single partition. We actually do dual partition so uh for uh resiliency.

D

But you know, a single sort of image is just 250 megabytes which, for example, allows us to basically netboot- and we can you know, potentially even not boot over the wireless connection or something like that right.

A

Is it okay to jump in with questions, or would you rather yeah.

F

A

Yeah so so a couple come to mind so you're you're landing on potentially bare metal or a vm that looks equivalent to bearing oh.

E

It's always it's.

A

Always bare metal, then they're getting in the root of trust and all that and um taking charge of updates to the os itself in a way that is brick resistant. uh Are you x86, plus arm agnostic as to what the actual hardware is uh or.

D

A

That's right. Yes, we are we're also.

D

Looking into risk five at this point, you know.

E

Okay and running.

A

E

Gpus, like nano boards stuff, like that, you know so it's it's definitely. Silicon independent.

A

And are you actually hosting a mechanism to also update the lower level uh boots, bootstrap firmware or maybe any supplemental firm, any firmware associated with supplemental hardware devices like smart mix?

A

I don't know, storage controllers whatever sometimes also have a need to get updated.

D

Yeah we we have, you know sort of. We have that architectural capability. We do some of it on arm. We don't do as much of it on intel, but you know there's a few customer requests. Actually you know they came from. You know, dell. When dell had the um gateways, you know product line right, but since you know they'll sort of gave up on its own gateways, you know that didn't really go anywhere uh but yeah once those requests come in. The architecture allows you to take care of the you know those levels as well.

E

Yeah and there's some things you know like bios update, for example, like some some users ask for it. um The if you have it's usually based on hardware. You know the way they do it like capsule update, for example.

E

If there's that, then you can certainly do it and you know there's ways to orchestrate that, but it's also depending on where these devices are sometimes not advisable to update things like bios, unless you really have to, because that's when you definitely, of course, can break the device.

E

So uh it goes wrong, but um yeah there's a number of different things: architecturally that can be done, but certainly evos, updating it any of the guest applications. On top you know and then, as as we said, it's it's it's designed to lose connection, you know. Basically the way it works is you you take evo s, you put it on a box, it taps into root of trust. In the box. You like a tpm. It creates a crypto based id based on that tpm or those credentials.

E

Now you can no longer log into the box anymore, because that's one of the attack factors where bad things happen when people just kind of you know log into the box and start doing things, you can then also use the apis to shut down any uh ports that you don't want access to like shut off all the usb ports, for example, remotely because it's bare metal um then that crypto base id is what you use to log into the controller. You know zedita, we, we are an example of some of them.

E

We have a cloud-based sas subscription that uses it, but you know anyone can go develop this and and we've seen other people. um You know looking at that because of that open api, there's you're not locked in any particular one, there's a basic open source controller in the project and we're looking at ways to kind of grow that, because everybody wins, if we make this the android of the edge. But at that point you could you basically connect power, networking bootstrap it secure dial tone to your controller and you leave.

E

Everything else can be done remotely at that point, whether it's deploying kubernetes or any other kind of combination of apps, also really important in the industry. So clearly everyone's kind of moving towards this. This clustering- you know you know, there's a big kubernetes trend, but not only is there a lot of windows out there, uh whether it's in manufacturing retail point of sale systems, you know hmi scada, yada, yada, there's a lot of industrial players, a lot of players in general that they pre-qualified a linux linux image and they don't want to change it.

E

You know this is my image and the beauty of this is that they can run it on top uh in a you know, kind of call it up. You were in a vm and the dr, even the drivers within it virtually pass through evo s and you've got the virtualized. I o and it's like evo, isn't even there, and so you know common paradigms that you would see in in certain data center applications. But you know really built from that ground up uh to meet that paradigm.

E

uh It makes sense and we can kind of get into some more details. But I mean.

F

In another sort of perspective, right depending on people, are coming at this if you're coming from the data center, you typically don't think about this, but the fact that there is you know baseboard management, controllers, management, networks, pixie booting right, maybe remote power control right there's. If, if things actually go haywire, you can always reset things and restart in in the data center right people that have deployed tanks at the edge they've had these things.

F

You know going back a couple years when that they did a demo in chicago and they had to go around the cherry picker. So someone could get up to the light pole and press the reset button. That was all they had to touch right and and basically the underlying eos. It's not only the security functions. It's also that sort of robustness function that you cannot. The device can always talk to something over the network that can tell it what to do, and it always gets back up right as long as the hardware doesn't break.

F

We can't deal with broken hardware, but but it's sort of building that infrastructure without assuming a separate network or other connectivity just using the same connectivity as sometimes.

A

So the diagram, where you're, showing kubernetes running uh on top of eve, is that hosted in a vm then or is that run? Is that a kubernetes cluster node that is directly installing a container runtime on evos itself, without a without a vm wrapper.

D

I mean by default, it's hosted in a vm, uh but we can basically because we're actually using container d ourselves right. You know, on the bare metal I mean we can potentially proxy that into the vm. uh Those requests actually never came. I mean we sort of architecturally made it flexible enough. You know sort of thinking that people would be asking for kubernetes and bare metal. uh Those actual requests never came, and you know I guess I have public cloud to thank for that. You know because they sort of conditioned everybody that vms are okay.

A

Yeah and it could be that the vm would be a a useful additional security barrier of some sort to protect eve, evo s itself from evils that might link in from the kubernetes world. If there were such things,.

F

E

And other apps yeah yeah yeah.

F

Yeah, if you want to deploy multiple apps, I think that was what you're about to say. It also protects in that direction. Right, multiple, separate apps. You have a windows vm sitting next to your k3s.

F

They they're isolated from each other, but also in in the sense that eve actually can provide some firewall and vpn functionality. That's external to the workload you're running the vm you're running. So now you know your vpn configuration, even if your application is broken into because it's talking on the network. Well, you can't break the vpn. So if the vpn is constraining it to only talk back to particular cloud provider, okay, it can't escape out of that right.

A

Does evos have a provision to allow things in the application level or application ring to get direct access to devices you know in kubernetes? Sometimes this comes around where an app running in a pod really wants to deal with. I don't know anything from a nic to some.

F

A

Usb hardware device- and there are ways that might allow you to not have the base os of a cluster node, claim it and pass it through.

A

There are potentially security risks involved with allowing it to, but you can put it together if you want to, is there a a way to do that with the os.

D

I mean there's two ways to answer that question right. You know. First of all, you know, since we are predicated on hardware assisted virtualization, you can do the device assignment to a vm. I mean that's easy right, so even that vm that runs you know, k3s can get all the devices you know as long as their pci devices, and you know even some additional devices that are easy enough to uh pass through like serial ports, and you know usb controllers and whatnot.

D

um So we have customers using gpus, obviously, and that works very well um and the beauty of it is that it's a true pass-through right. You know, unlike what I've seen in some of the proposals uh around you know, container ecosystem at that point. Eos is just completely hands off. You know when it comes to that device right. You know the vm that you've built for your k3s.

D

You know deals with the device driver and that's about it. uh We can do one better. We can actually do you know full-fledged virtualization uh and it's helpful when you actually have proprietary device drivers, which is you know a lot of times the case on platforms like you know, jetson, you know from nvidia uh because with jetsons, for example, you end up getting a linux kernel. That is really, you know, incompatible with some of the kubernetes requirements and there's nothing that you can do about it, because that is the only linux kernel that would support.

D

You know the drivers from nvidia. Now in eve, you can actually put that into a dedicated vm and then virtualize out of that vm uh and essentially have what's in zen. You know land is known as uh driver domains, so that it, you know it's a vm that hosts just a particular driver and nothing else.

D

E

Now other questions are, you know, kind of about the overall approach.

E

Yeah, so um you know part of the reason for reaching I just kind of wanted to learn what you guys were doing. We are um you about to ramp up uh you kubernetes messaging. You know fyi uh middle of kind of third week of april we're we're starting to kind of announce. um You know, support you know, kind of that bridge to uh kubernetes with eos um we're working with some providers like like uh you know, ranch are now part of souza.

E

You know in terms of their managed commercial run times, you know being able to run on top of the os. um Of course it could support any any sort of open source effort. um So there's just a lot of activity. That's about to spin up, um it's kind of more and more people are like hey.

E

I want to use kubernetes at the edge, but there's there's unique challenges, as you guys know, and so, if you want to operationalize it, you need the right foundation, um we're also getting evos baked into other reference architectures within the industry. You know, for the same reasons you in terms of that that the universal approach uh either stuff like around oil and gas, the open group um we're working with lf energy, some other consortia. um You know just again, it's all about. You know everybody wins.

E

If we get to a more common foundation, lowest common denominator, and then we kind of bridge paradigms.

E

F

E

We'll send you the stack um and you know, kind of, let you guys chat and, however, we can help, but uh we but we'd love to create a bridge to the work that you guys have been doing. And um you know cnc more broadly speaking, because we think there's a lot of value to help bridge that kubernetes story out into the field.

E

James also from a red hat perspective, we'd certainly be interested in talking about potential to make some bridges.

E

Any other questions, thoughts last call all good.

B

This sounds good.

E

And just you know so as a community we're we're approaching 60 unique contributors, you know good growth curves, you know obviously zedita. We contributed the code, but you know the measure of any good community.

E

Is it's not just you uh contributing to it, and so this year you know not just the kubernetes there's, there's increasing arm and gpu support, even though it's silicon agnostic, it always gets you a little tricky when you get into those the fragmented arm eq system, but um you're always looking to optimize the footprint uh and then uh you know other industry efforts. You know between um you, know, cncf and beyond, or linux foundation and beyond.

A

Maybe this might be later in your deck jason, but maybe if you can give us some clues on how we would get started like suppose, I wanted to just try this out uh proof of concept on my own hardware. Where would I go for a getting started thing to run it on? I don't know, I assume it might run on a pie or I don't know.

F

Yeah we have a win like.

A

X86 and uh maybe fill us in on whatever community meetings and slack channel and ways.

E

To go, aaron aaron can help with that. He dropped some stuff, I think, into into the thing, and we can follow up uh with that. You know aaron if you could follow up with that um yeah as any any self-respecting open source project. You know there is a uh an image for a pie um readily when you, when you grab an x86 box. Generally speaking, you just take vivo s, you get it from github you plug it in and and it's gonna it's gonna run.

E

Generally speaking, you might run into issues you know, because, like cell drivers, you know always a challenge in the linux world because it is rooted in linux, but it's got the embedded hypervisor watchdog, timer stuff, like that, but just foundationally it'll run so whether you get a raspberry, pi or a noke or otherwise.

E

But again this is the benefit of creating an ecosystem around it. The hardware providers are starting to even upstream drivers for it you know, and so that's the ultimate goal is like literally make it a factory os, because it's a universal option and then you don't have to think about it so to get started. So it's it's, you know we can get you hooked up with. With the um you know, the code in the in the the github and some getting started stuff uh fyi.

E

So there is a again the open source controller within the project um we're about to launch. uh You know from a zadeta standpoint, a dev program um where you can actually get to the cloud ui that uses those open apis.

E

That's makes it even more usable and we literally have k3s you know in there and you can just click a button and deploy it, and so we're going to do invite only alpha version of this developer program here shortly. So, if you guys, you know want to get access to that. You know. Let's talk because then you can kind of give you the easy button. um You know as part of that, invite only phase and then we're going to make it automated in the future.

E

But again, that's that's just because most developers do not care about security management, upfront they want to get to hello world and some app and what we see all the time is people like. I don't understand what you what you guys do. I don't understand the value of this or whatever and then after they actually try to deploy something in the real world. They're like okay.

E

Now I really understand it, and- and so that's why we want to get people access the ability to get bootstrapped develop your app, you know, do whatever you need to do, but then don't have to reinvent security management after the fact, because this goes down the silicon measure boot remote station, all the bells and whistles yeah. I I think this person in particular.

A

Would be interested in kind of that you know directed path for a proof of concept that would take yeah.

F

Into a kubernetes.

A

Hosted version, obviously, since this is a kubernetes thing and maybe with the assumption that they didn't have to think about it, but it ended up kind of default. Secure out of the box.

E

Yep yep, that's the whole goal. I mean this is this is set up uh especially I mean you know again.

E

Anyone could build the controller, but the way the way that we've set it up is like you, don't need to know a lick about it to deploy a box, and just you know out in the field, because usually it's a very different skill set, as you guys know, once you get out of the data center, so um yeah, so we'll get you, we can get you kind of set up with some of that and and sort of build it.

E

As you know, a poc, um you know again it's sort of like that early alpha dev program and then we're going to do sort of like this rolling thunder as a community bridging to other kind of popular application communities. So, first, it's kubernetes, that's going to be the first push to the broader kubernetes community, hey guys, if you're, actually, if you're deploying kubernetes out in the field, you're going to need some sort of foundation to make it secure to be able to manage the hardware and all that so that'll be phase one.

E

Then we'll start attaching it to other things like you know your tensor flows in the world and- and you know other types of application stacks. You know the ejects and pledges and stuff like that. So um but but kubernetes being the first priority, so yeah we'll get you set up I'll, send you this deck, we'll get you guys set up. You know um I'll just send it, but you know um I don't know roman and eric any any that that work for you guys just kind of get the team hooked up.

D

Just one last thing that I wanted to point out is that uh lf edge is actually a place where we host akrano and ikrano is in business of producing these blueprints, which is basically industry, approved. You know, approaches uh to various use cases, so there's a few blueprints. You know involving kubernetes, obviously, uh and how it you know, gets deployed uh in various. You know edges that jason talked about.

D

uh It would be interesting, I mean to sort of, maybe for you guys to kind of like take a look at those and see what you you know, think, or maybe there's some of the cross-pollination between two of the communities, because you know it's sort of always useful for the other side to know it's like what the other side is doing.

A

Okay, just a time sink warning you can have about five more minutes same to anybody with questions, go jump in and get your questions in now.

F

One thing I wanted to point out before it closes is yeah aaron posted some some links in the chat for this, and once it's over those, the the chat will disappear. So if you want to click on those now, it would make sense yeah in terms of how to hook up with the eve community and pointers at the slack channels and whatever in our meetings.

C

I also put the main links in in the the document um that it gives the interrupt for you guys um also what uh to kind of uh go on to what uh roman was talking about with arcano.

C

um There is also some work uh trying to implement uh cube badge, so that could give you um another reason to to go into that, and so just as a fyi.

C

I wasn't saying if you guys have more questions about eve or you go play with it today. Tomorrow we do have open office hours, so um they're at 8 am pacific.

C

So please join us. I think I put that link in there too, but that's pretty easy to find and you can always. If you have any questions, you know shoot us all an email or go on our slack.

F

Office hours link into the agenda.

A

Notes, if anybody's watching this later on youtube.

E

And I'll drop a I'll drop a little bit of a teaser since we're going to do this kind of rolling thunder thing around uh it's on april 22nd, it's you know, part of the again. The reason to reach out is hey. Can we help at all like just in terms of that foundation?

E

But you know, as we kind of roll this out I'd I'd like to kind of make it known and suppress that we're collaborating, and you know helping build out the vision that you guys have been growing and all that so just something to think about. But you know just as a okay.

A

And feel free to promote that I mean we allow anything on open source projects, not yeah.

F

E

Yeah, it's not a.

A

Sales pitch channel but go ahead.

E

And use a slack.

A

Channel to remind people.

E

We would yeah, we would have to keep the we would keep the commercial component. You know separate, um you know and maybe do a blog or something but yeah definitely aware of the sensitivities around that cool. I gotta run as well, but um you know thanks for the time we'll send all the stuff off. If anyone wants to tinker, I mean, actually, you can go to github, but if you guys want to get um you know access to some other stuff. You know like the the easy button side. Let us.

A

Know: okay, thanks for doing this, it was a great presentation. If there's anything you didn't have time for or think about later feel free to just throw it in the agenda. Notes document for links later, particularly things related to kubernetes integration.

A

The our meeting uh in two weeks on the apac europe cycle tentatively has a speaker on a new project called open yurt. I don't know that much about it myself, I'm looking forward to it, but I get the impression that this is something along the lines of a cube edge. That is a extension to kubernetes, to support some edge use cases, and it is now recently became a cncf sandbox project, so that would be on the april 7 meeting for those not familiar with this group.

A

We alternate time so this is the north america time cycle which comes out to be 9 a.m. Pacific.

A

The alternate cycle was done to make it easier for membership in asia and eastern europe and that one in my time zone time tends to be 10 pm or 9 pm at night, depending on how daylight savings time intersex does anybody else have any last remarks? Questions comments.

A

Okay, I'll take silence as no and uh we'll declare this meeting over I'll end. The recording and thanks everybody for coming. If that open your presentation on a new kubernetes extension for edge appeals to you, maybe we'll see you in two weeks and thanks for coming.

E

Thanks again for having us we'll catch.

F