Red Hat OpenShift Detroit 2022 OpenShift Commons Gathering, 25 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: MicroShift on the Edge - Red Hat OpenShift Commons 2022 Detroit

Description

MicroShift on the Edge
Red Hat OpenShift Commons 2022 @ Kubecon/NA
Detroit, Michigan
October 25, 2022

Speakers:
Jon Cope (Red Hat)
Frank Zdarsky (Red Hat)

https://next.redhat.com/project/microshift/
https://commons.openshift.org/gatherings/kubecon-22-oct-25/

A

B

My name is Frank Starsky I'm, the senior principal software engineer, I'm, leading the edge Computing team in the emerging Technologies Group of the office of the CTO, and my team is actually the one that incubated microshift and with me today is hi.

C

I'm John cope I work with Frank and worked with him in ET and up until recently, I've transferred over to the uh uh openshift engineering side of things following the micro shift into productization and I'm. A software engineer.

B

All right so I hope you didn't miss it this morning, when Reddit uh sent a print press release announcing the new Reddit device Edge and that Reddit device Edge actually contains microshift. So this is why I want to talk about this today and before you know, uh John talks about the details of microshift.

B

Remember the talk of a lucky team right, met and Ian earlier today. Right! That's! You know we are collaborating with that team. We were collaborating with the team and we're continuing to collaborate to actually make microchip um possible and also meet our customers use cases, because that's the model um that we typically use, and so let me unbox a little bit on what is um related device Edge.

B

If you're looking into Edge Computing use cases with openshift you've probably seen. You know that, for example, BMW in in Germany does Edge Computing, where they're doing drive simulations to train their autonomous cars models and openshift clusters right. You have single node, openshift deployments with Verizon running 5D base stations, but there's this whole class of use cases that we did not tap into and that's what we're doing with Reddit device Edge right as this use cases that you see here right. It's the the drones that you've heard about earlier.

B

It's about satellites, so Aaron uh Boyd this morning mentioned that Microsoft is actually in space right now on a satellite we have use cases with iot gateways and these uh very small 5G radios, similar to the one that you see down there, and why are these types of use cases different? Well, that's because um we're talking not about servers in a controlled environment, we're talking about field deployed devices. What does that mean right? We're talking about devices and they're embedded compute unit, and these computer units are not server Hardware.

B

There are single board computers and systems on chip and you buy them because they are optimized for certain class of applications like running machine learning, models like doing signal processing like doing you, know, media processing, and they come with all the I o that you need. They come with all the accelerators. You need whether GPU TPU networking accelerators video encode decoded accelerators, but they are rather Limited in resources right. So you get like eight 16 32 gigs of RAM small number of CPU cores and unlike servers, they are then not extensible or not really extensible.

B

um The other part is that um a lot of that Hardware actually does not so support a lot of the highly standardized hardware and firmware features that you would find on server grade, Hardware right, they're, implementing it differently and that's a problem for the operating system, where they're trying to light up the components of that Hardware.

B

The other aspect is comes from field deployment, so you can imagine in these use cases that you've seen that um there are additional engineering constraints in terms of power. You need to consider battery power, solar powered power being unreliable, failing at any moment in time, or that you have constraints on the thermal design power right. You don't have cooling, especially like in space or networking constraints. You may not have any network at all. It may be intermittent, it may be slow.

B

You may pay for it by the megabyte right if you're thinking about satellite links um it. It is not a network environment that you control, unlike most server deployments and finally, of course, security. What happens if you don't have any physical Access Control to protect these devices right next challenge is um the expectation of the deployment model.

B

Ideally, what we want to do is hire a suppliers send directly on site rather than to your central it where you install something and then ship it out right, it's more like the Wi-Fi access point that you buy, you unbox it you plug it to the wall, you plug it into power of our ethernet and you walk away right, that's the expectation of the provisioning model and finally, on these devices, you don't have any out-of-band management mechanism right, so there's no ipmi, there's no.

B

In most cases, no human that can actually fix stuff when they break there's no management Network, and so these are challenges. Another challenge is that a lot of companies from their systems embedded system, space, they're used to very different tool chains like building, um basically the whole system from source using yokto or built root tool chains right so very far away from containers and kubernetes and all that stuff.

B

Okay, so let me unbox the rated device Edge. What we're actually doing is we're releasing a Reddit, Enterprise Linux, surprise right, um it's an Enterprise Linux with a small footprint and it's not based on a package base, but on an image based model plus it contains several new features and tools developed for the device. Edge right, of course, reusing some of the features that we see in other flavors like real chorus.

B

It's image based. Why? Because think about your phone right, you're, not downloading individual packages. Vendors are creating an image of the whole system operating system, kernel applications baking it into an image, the new version. You build.

B

You test your role, the whole system out as a unit right and, of course, what we want to do is make it easy for you to roll your own operating system right, build your drive, the drivers that you need into this leave out other stuff, add an agent and for that there's tools like image, Builder that allow you to do just that. It's also designed for ADB deployments. So what happens when you download an update? Obviously you want to what you can do. Is you download it into a parallel file system right?

B

You stage the the data and that moment Network can fail, but your updates will still be safe, because this new system is in parallel. You then just basically flip a pointer and then reboot the system, and then you automatically come up with the new version or you flip, the pointer back, you reboot and you're back at the old version right. It's a AV, parallel system and that's very efficient with Delta updates. They have what's called Green, Booth green boot health checks.

B

So, like you're used on kubernetes with your containers, you define health checks, telling you telling the system did it come up healthy and if not automatically roll back. That's the fighter device on board, which is an industry standard, that's basically ensuring that you get a late binding between device and owner you plug in the device it discovers its owner. It will onboard with the illness management system and then, basically, um you know receive its initial credentials in identity.

B

And finally, this red advice has been built for RPM container and VM workloads, plus it contains microshift. So you can also do um kubernetes workloads in the same model. Why would you want to do that?

B

Well, we've heard it already from several of our customers this morning right because you start with podman lots of tiny little pots right and but, as you grow it's going to, let you need orchestration, then you may want to start scripting something, but that's not a good idea right, because kubernetes is the de facto like industry standard for orchestrating containers, or you may be standardizing on the kubernetes model for software life cycle for deploying you know, off-the-shelf kubernetes workloads or you may want to standardize across cloud and the farthest Edge, your your software development and operations and potentially, of course, you want to grow right to to more nodes or to ha and with with kubernetes.

B

You can do that. Obviously you don't know this so again in the Reddit device Edge, we include microshift.

B

What is John will talk about, but the key Point here is: it also supports different models. You can use it the real device as a container host and dynamically deploy and undeploy and change your workloads on top or you can bake your container applications into a system image. It's like on iOS or Android. You have your applications delivered by the vendor, your core applications and others. You are download from the App Store and those are the models we want to support to you.

C

All right, my notes, are about font size.

B

C

So I'm going to do it off the sides, so this is where openshift comes in. As Frank said, users need to be able to run containerized workloads at the edge in an orchestrated manner. Ocean ships is great for running Orchestra or for orchestrating containers, but it's good for doing it in a data center on bare metal virtual machines, uh but it's not as great on a resource constrained device. That's where microchip comes in.

C

We took openshift we uh or rather took all the best parts of openshift, all the ones that were pertinent to the edge, uh condensed them down and turn them into a single deploy or single binary Deployable as an application on top of the operating system.

C

I'm trying to remember my notes.

C

So one of the key things to remember is that microshift is openshift, it's not a flavor of openshift and it's not uh a sort of side variety of it. It's built on openshift bits. It incorporates uh core openshift and kubernetes components inside it. uh So what I mean by that is?

C

Oh I, apologie I'm, getting ahead of myself uh before I, get into that. We also, um we don't just mirror openshift into microshift. We build micro shift in the same pipelines that would build openshift and we test microshift in the same pipelines that we test openshift in um this to us, was critical early on and it it paid off in that we are now able, to. You, know, keep Pace with openshift releases and uh catch regressions and uh incompatibilities early and fix them quickly.

C

A second thing that we are benefiting from from the openshift CI is that we were. We need to be able to rebase onto the openshift bits uh quickly at the same Temple that they release typically nightly uh to follow the Z stream releases.

C

Why stream releases are more complex and usually require some Hands-On by an engineer to handle the rebasing uh Frank wrote uh a very complex automation, script that can take in a version of openshift pull down the ocp release, image dereference the the actual commits further, the each openshift component source code and then rebase microchip back on top of that, so we'll vendor in uh Core Business of openshift and pull down the uh container image digest for the uh small set of service or cluster services that we host. On top of slide me, please.

C

So this is a bit of a diagram of what this looks like. Given a z stream release, we will pull in the openshift source code and vendor it independently, discreetly per each component that covered in the Microsoft binary in the bottom. So all that's compiled into a single binary run as a single application.

C

uh We also pull in these component manifest which we embed as binary data, those those manifest Define. Our cluster level, Services uh noted on the right here: service, EA, openshift, DNS, Etc I'll, get into those a little bit more later and then we're able to uh quickly build that and turn it into an RPM which can then be turned into an uh RPM os3 layer and deliver to devices.

C

So speaking more to the the architecture of microchip, we spent a lot of time trying to redefine the the openshift Paradigm of high ability, High, availability, um scalability and the ability to just maintain. You know thousands of workloads at a time. What we came up with was a very small binary. I think we said that roughly half a gig um and inside that binary we execute SCD the cube, API server, Cube controller manager, the scheduler, the cubelet, the um from openshift.

C

We we run the route controller and the policy controller, and uh a couple of specialized controllers that we wrote well I, say controllers, but uh services that we run to um uh ensure that we can deploy additional crds and our cni and CSI plugins.

C

In addition to this, so uh because Microsoft runs as an application, it stores its cluster State on the.

B

C

Machine not in config Maps or in dynamically provisioned volumes on you know, access through PVCs. This gives us a couple of benefits. One is that we can continually restart microchip without damaging the cluster or interfering with runtime applications.

C

This is really useful when combined with our support for offline images and offline manifests in particular offline, manifest can be delivered to directories that we've defined in a microchip where our user will place their manifest as a customization or sorry customize, yamls uh and then on start microchip can pick those up and deploy them on the cluster right away. This means that you can deliver them as a OS tree layer after deploying microshift or at startup, uh reboot, microshift and within seconds. Your workload will be deployed.

C

Combine this with.

C

Like over-the-air updates from fleet manager and you're able to push out workloads from the central location to these devices and be confident that the cluster will start and if it can't start if there is a greedy workload that crashes the system or uh is just now configured, we can rely on green boot uh to roll the system back and preserve the cluster state and and maintain the workloads I will I. Do want to note, though, that green boot will not save your workloads. If your workload is uh broken, it will have to be repaired.

C

It's just not something that green boot does uh slide. Please.

C

uh So packaged with uh microchip we've included a CSI driver and a cni driver, and we spent some time but a significant amount of time researching these and really trying to narrow it down. There's a lot of variety and a lot of solutions out there and based on Partner feedback. We landed on uh odf lvm by red hat, which is the operand for the lvmo operator, the lvmo or sorry odfl.

C

Vm is really valuable from a a deployed device standpoint, because uh thanks to uh deploying uh Rel underneath it we have lvm built in by default. This means we can protect the operating system from greedy workloads that get out of hand writing data to the the system, and you know ensuring that they, they can't crash out the the device by just dumping, all their data. uh Secondly, we also protect them from each other, so a workload can't interfere or access.

C

Another workloads data without uh you know, Express configuration from a administrator and uh the the last thing that that brings with us is the ability to well odf lvm comes built in with the snapshot, support and is capable of preserving enough space on disk to store your snapshots.

C

uh Guaranteeing that well ensuring that uh workload requests can't starve out any support for snapshotting.

C

On the networking side, we opted for OV and kubernetes, which is a Upstream version of OBS, which is uh another red hat component and well supported and well used. So it's a high quality highly trusted uh technology. It also ensures workload, isolation at the networking layer and uh satisfies our partner needs for multicast and UDP.

C

So, with a minimalized openshift, what we're down to now is a container orchestration platform, a security workload, isolation.

C

Safe update and rollback procedures, they can run on two two cores: two gigs of RAM and just 16 gigabytes of disk space, which leaves plenty of plenty of overhead for uh user workloads and we've. All we've also done this without sacrificing the pertinent and important parts of openshift, which is uh at the device Edge security.

C

So we still include a the controller for a security context substrates and we still support routes and Ingress I'll. Note, too, that those are the only openshift apis that we incorporate into microchip.

C

That's because microshift is very narrowly defined for the edge we made. The we made the choice early on not to try and solve every problem with microshift microshift is not designed to be a CI CD, uh mini node or a desktop developer environment. It can be used as that, but that's not something we support and to that end we were able to remove a what we were able to narrowly define the scope and, let's cut down the size, pretty significantly.

C

So the next thing is, how do you manage hundreds or thousands of headless devices deployed in a factory or deployed out in the field, often installed and maintained by non-technical personnel and perhaps in remote locations? Well, openshift would have solved this internally. It manages its own system and manages its own machines.

C

It's capable of provisioning cloning and and uh transferring workloads between nodes microshift is a single node operation and does not interact with the host system in terms of control it doesn't set, um it doesn't specify system, configs, machine, configs and most notably, which I shamefully almost forgot does not deploy a red hat operators. There are no operators in microshift, and that was a deliberate decision early on as well. There's no need for an operator if your machine is a single node.

C

If it's already got a life cycle management at the system layer- um and is you know it can be managed remotely with an atomic operating system so to manage these devices, we are leaning on more tried and true red heck technology.

C

At the system, layer fleet manager will enable users to push out workloads from a singular location out to uh you know, one-to-end connected devices uh and be sure that those are either gonna uh perform an.

B

C

Or roll back, if they can't report otherwise at the kubernetes layer, um we're able to also push out manifests directly when connected to ACM, barring, of course any apis that we don't support. So this is typically your core kubernetes workload, definitions.

B

C

There's a lot more to cover, but there's there's.

B

Simply not enough doctor that yeah, so, just to summarize, you know um I I, hope what you take away is that this device Edge is a very different class of edge, Computing use cases and that with red device Edge, we actually want to expand where you can use regular platforms to this class of use cases and the way we do that is actually addressing these challenges that make the device Edge unique through an operating system that has all these features, enabled and changed to an image based model right, because we believe that is that this is a Time proven way actually of operating uh large numbers of devices and with microshift.

B

We basically said: let's not reinvent um a lot of stuff, like the operating system shall do we just rely on these rail capabilities and then layer on Top, This RPM that you can embed into your S3s. If you want, you can embed your workloads into those OS trees and then treat everything as an image-based model, and just very briefly, to clarify that uh it doesn't get misunderstood when John.

B

When you said, operators I just want to qualify that the operators that manage the operating system and the configuration or server-specific Hardware or basically a um a multi-service cluster, obviously your application operators would still continue to run so just wanted to make this distinction right. Thank you very much and I'm not sure we have thank you.

A

Testing there you go, we do have one question from the virtual audience: I'm going to read it off and see if I get all the acronyms right does microshift have a load balancer, and can you allocate at SVC with a type load balancer? Does that.

B

B

So right now you know until 4.12 comes out and we can actually officially raise the the RPMs for for microchip. You still, of course, have the option to go to the site to build stuff from Source. We also provide a you know, Community built RPMs, for that in the current version that you download. No, we do not have the inbuilt service load, balancer you're, using node ports for that or you're deploying the metal lb load balancer. On top that's a very simple operation.

B

It's documented, and then you get the service load parts we do intend actually to um provide a slightly more Compact and smaller footprint version of load, balancing that one of our microchip customers asked us for so that will also be available, but not in the current version. You.

A

Can download not yet but soon, okay, awesome! Is there a question in the back here, hang on a sec, all right.

B

It's part: it's part of the core microchip, that's basically running as a systemd unit. So you say you know, system control starts microshift and will automatically.

A

B

Start like the whole control plane, including hcd, and run that on the host and what you see running as pods are those hosted Services, which we will partially make also optional, like of NK top lvm, the openshift Ingress or the router, um a bunch of DNS and the server CA. Those are deployed on top of memberships. Yes,.

A

All right is there any other questions all right, I know here here: can you use the microphone just for a sec just so it gets recorded thanks.

C

Shirts on a single load, open shift.

B

No, it's not so you're, not open shift. Well. Openshift has a couple of deployment topologies right by default, you separate the control plane into dedicated machines from the uh from the workers. There's a topology called free, node compact classes, where you basically co-locate those in order to just run on a free, node, AJ footprint and then there's single node openshift, which is basically exact your openshift, but just running on a single machine. It also has additional tunings to basically constrain the CPU footprint to a single core, but all of that is playing openshift.

B

That means you can use exactly the same processes to manage the workloads. The cluster, the operating system and configuration top down right. Microsoft is a very different model because we're basically using a different flavor of Rel. If you want not real chorus, but but this well for device Edge, let's build towards the device Edge model, a microshift is an RPM, but you basically dnf install microshift, and then you have your running cluster. The consequence is that this Microsoft cluster does not manage the configuration because in this device world you want the system.

B

Builders want to take control of all of this operating system right. This is what um what uh Colin early on called like exposing you know all the the linuxy part which openshift conveniently abstracts from.