Red Hat OpenShift OpenShift Commons Briefings, 24 Aug 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OpenShift Commons Briefing #92: KubeVirt - Kubernetes, Virtualization, and Your Future Data Center

Description

Kubernetes is a great orchestration tool for containers, but why stop there? Containers and virtual machines are going to coexist in the data center. Let’s re-envision our virtualization and cloud solutions with Kubernetes as a single underlying platform.

In this briefing, get an introduction to KubeVirt – a project to converge the future data center using Kubernetes as its infrastructure. We cover how we are implementing a caring and stateful environment to run pet VMs in containers on top of Kubernetes – without contradicting its core assumptions. We also discuss gaps and how we plan to tackle those, drawing on our experience with KVM and caring for pet VMs (and cats) for many years. The session also includes a demo of how we are doing this today and where we want to go next.

A

Well, hello and welcome again everybody to another open ship Commons briefing. We're really pleased today to have folks from the Qbert project here to give us an overview and to talk about what you can do with it on kubernetes and I'm right now we have it mark I'm and Fabian Deutsch and I'm gonna. Let them talk for about probably half an hour, do a little bit of a demo and we'll have live Q&A at the end.

A

If you have questions, ask them in the chat, we'll try and answer them will read out the questions at the end, so that everybody who seems we're recording afterwards can hear the questions and we'll have some live Q&A as well, so without any further ado tomorrow. Let's, let's take it away and get started Thank.

B

You Diane so hi everyone, I'm Emma hi, my run. The engineering groups for distant virtualization and container management in Red, Hat and Fabien Deutsche was presenting with me is leading the engineering effort around Qbert. So what is Q Bert? It's an upstream research project or we're looking at. Can we use kubernetes for converged infrastructure and when we're saying converged, infrastructure, we're talking about containers and virtual machines.

B

Now this is really early days, but there are some interesting concepts that we fought for sharing because we're looking at interesting aspects of how kubernetes is doing things and leveraging different mechanism in kubernetes that are pretty advanced around extensibility and trying to map different concepts from a the experience that we have from. You know, scheduling other workloads like virtual machines on cluster to the kubernetes world. So we fought it's and while it's still, research is interesting to share, and hopefully you'll do the same.

B

So this is a background when we talk about virtualization in in the Linux world. We're obviously talking about KVM, and the nice thing about KVM was that it made a virtual machine just the user process. So we just get to Amica vm running as a user process, now just having one virtual machine, the factories user process not good enough to stop as a solution as a technical solution, but not a full-blown solution, and this is where a project like over it came along.

B

You focus on I want to run a group of virtual machines on a cluster of hosts which are in storage, and you know, full-blown management, user interface to orchestrate hosts lifecycle, virtual machine life cycle live migration law, balancing concepts like that so and enterprise. You do data center virtualization with a VM, then OpenStack came along and tackle infrastructure as a service, so more. How do I provide api's to virtualize, not only the virtual machine, but also my storage, my network and other services, but both overt and OpenStack focused on virtual machines.

B

As the court ended, the OpenStack is looking at other things as well. A virtual machine were really that the start in the corpus and then came containers in kubernetes now kubernetes is dealing in a similar area of I want to schedule or closed containers on resources, fostered resources, hosts storage network and the nice thing about kubernetes wood containers. It's a bit more of a generic use case than virtual machine wash.

B

So the question was: can we take this generic use case that kubernetes handles and look at virtual machines as well, because hey virtual machines are just a user process, we can maybe put them in containers for examples, and virtual machines already share some concept that containers use containers, use, selinux and c groups for isolation. They also use namespaces, which is not relevant for virtual machines, but really same concept. So can we look at this and get to it? Convert infrastructure we put on kubernetes virtual machine and containers just get together now?

B

Why would you want to do this? So, yes, containers are cool containers are all the rage. Brunette is a spray and with things like open ship, it actually changes more than just scheduling war, close act which any of your DevOps, that your development lifecycle, how you approach even software, see I it's a broader solution than what virtualization on its own and to solve. But when that there's a huge investment in virtual machines, they're, just not going away anytime soon, just like on-premise versus cloud isn't going away anytime soon.

B

So, if you think about it, we still have mainframes with us, because the investments so large the virtualization market is still growing. So, if you're investing containers in five or ten years from now, would you want to have multiple silo deployments or, if you like, to have a single infrastructure that you can put all fewer workloads on it and think about how advanced it will get to be at that point and how easier it will be to maintain and operate and obviously would use cost if you can do it on a single infrastructure.

B

Another aspect is, as you have all your applications that used to run in virtual machines. You want to start moving them to containers, so you have the database and the front-end portion of your application. So maybe you start with the front end and you move it to containers if you can move it to containers and it still runs on the same host and get the local affinity that the container is calling the database, but it's on the same host so known that will plate and see everything is simpler to manage and control this way.

B

So that's another benefit. As you migrate, you don't need to plan for different Hardware footprints, because you're now migrating more clothes from the virtual one to the containerized one. The workflow can stay on the same host now for virtual machines. We can benefit from some advanced concept. That kubernetes has so virtualization is very maturing. Has a lot of things that kubernetes still is still catching up with, but kubernetes is first of all, moving very fast and also brings some. You know newer concepts.

B

So if we look at daemon says: oh that's a nice feature, we could get it out of box in kubernetes and get it for virtual machines. Suddenly rolling upgrades things like that or things that we could leverage there as well.

B

The benefit to kubernetes is that there's tons of experience on the virtualization side and tackling similar aspects, the one kubernetes as tackling, especially when you're looking at on-premise and Burnell use cases, though things like host lifecycle things like device pass-through for all of those there's tons of experience, and this is why we think this is going to be a win-win if we can look at inverting those solutions into one now, when you're saying virtualization and containers and kubernetes sometimes there's confusion. What is the use case?

B

You're talking about so I want to separate between two use cases. One use case is I care about containers. I want to run. Containers I want to run them in kubernetes, but I am still not as sir than that containers give me the level of isolation between tenants and workloads that virtual machines give me so I want to run my container inside the VM just to use the VM as wisely and that's something a project like clear containers is trying to tackle and providing better isolation for virtual machines.

B

Now, Cubert by looking at virtual machines, can also lend you know a solution to that problem. But that's not the focus the focus is. We want to run full fledged VMs and leverage kubernetes to do this. So it's not about the isolation story. You might have heard of its. We have EMS 10b ends, we have in overhead and OpenStack etc, and we want to take them and run them on kubernetes with all of their features, though I care about running a VM as the use case I'm a VM admin.

B

This is the over eat dashboard and it's not about running kubernetes with over things like that, although we just published reference architecture and running open ships on top of virtualization, but that is today when you use virtualized infrastructure to solve gaps that brunette is doesn't deal with on host lifecycle.

B

Hopefully, in the future we can look at a single infrastructure we're using over here just to showcase features from virtualization and some of our thinking, though virtualization same same concept, third set of note, storage and network, and we want to run workloads on them in this case virtual machines. Now some aspects are similar between virtual machines and containers or pods, and some are different, so we're trying to map them and try to look at those that make sense for containers.

B

Let's focus on enhancing kubernetes to close those gaps, those that do not make sense, there's no point bloating kubernetes to do everything in the world, rather relying on the extensibility of kubernetes and separate the aspects that are not core to kubernetes and just solving them for VMs and Fabien will later showcase in more detail the thinking around there. So things like memory sharing might be relevant to containers, because, if you're running containers, those are technologies, we invented for virtual machines but are relevant for containers as well. Ksm will scan processes.

B

If they have similar memory pages, it will merge them to conserve space memory. Ballooning is a technology more specific to virtual machines, probably not relevant to containers in kubernetes of this one, probably not relevant, Emily overloading scheduling all those aspects are relevant and we're. Looking about at scheduling, the kubernetes has scheduling a pluggable scheduler and you can extend it. It's not at the maturity that OpenStack is where 8 is different. Scheduler were implemented for different use cases, but it has the potential and the capabilities it just need to.

B

You know to be in the field more in those different use cases. Another aspect we're looking to solve, for example, when you're talking about running your own host. Is you want to conserve power, though in virtualization world you'll try to consolidate the workload say overnight by live migrating workloads into fewer hosts and shutting down the other hosts? That is a concept that could be relevant to the container world as well.

B

It's not all about just scheduling, resources or clothes on host. A lot of the issues you deal with is actually error handling, and this is where things like fencing comes along. I no longer see a node that had paws running on it can.

B

Does that mean the note is down or no I can somehow that that it's still running, despite not having a network ping to it, maybe through a storage mechanism or something else before I fence, it is do I, don't see the node or I, don't see a lot of know that made it problem is on my side now. Why do I want defense? I know it is because I want to make sure before I start that pod, that orb load again on another post. That is not going to prop my persistent volume.

B

If it's too prosperous databases, though some storage support exclusive access, some dome, if we're talking about leveraging existing storage technologies, for example, investment in them, it's something we would like to solve. They're different fencing aspects, and you start by soft fencing trying to SSH and restart the cubelet. Maybe before you shut down the entire house then go to IPMI then go to the actual power interfaces to turn off the power off that host. Maybe you want to check if the host is K dumping, if it has a terabyte or two terabyte of RAM?

B

That's going to take time, and you want that dumb for troubleshooting, so you want to detect its dumping. You now know the workload is no longer running because it's dumping, but you don't want to turn the power off yet now. Those concept might sound strange to you if you're focused on cornetist running on existing infrastructure, virtualized or cloud, but if we're looking at converged infrastructure, we will want kubernetes to be.

B

We have concepts like this or augment it with solution managing the environment, to give us those features that we get today for free from kubernetes, so we could get to a single environment.

B

Another aspect is network management. Today, kubernetes networking is sophisticated on the multi-tenancy, but not on the physical layers. Maybe you have different physical networks than you want to map different networks. So that's an aspect of hosts lifecycle and network management that we would like to look at and then we get to the you know the actual virtual machine.

B

So if you look at CPUs and memory, okay, that map's one to one and but we don't need to look in kubernetes if things like spice remote, audio video, that we do for virtual machines, because you know the concept is not as relevant for container. So, there's no need to look at that and our goal again is not to push everything into kubernetes, rather we're looking at what would make sense and where would each piece make sense?

B

And this is where some things are relevant, we're starting they weren't in the beginning, but hey Numa and CPU pinning are now in kubernetes. Live migration exist in virtual machines for years being talked about from containers may or may not be relevant at some point, but hey. Maybe we could look at those policies and do something about what the experience we're bringing from virtualization. So that is just to give concept of. Why we're looking at this and what we're trying to accomplish and now Fabien will deep dive a bit more into you.

B

What we're looking at.

C

Yes, so hello first, and let me try to share my screen. I hope that everybody can see my screen now, yep very good. So let me pick up pick up where a team are stopped, so Itamar showed us what we want to deliver on top of communities and with those requirements we looked at. So what are our goals? What do we want to deliver? And one of our I mean the foremost goal is to deliver a feature complete virtualization API, which allows us to live, deliver all the functionality we have today in its granularity.

C

So with all the details, we with spice displays and women were pinning and to do this in a way which doesn't contradict with with containers or kubernetes itself by the way chewbert is aiming to deliver an API. So it's not about writing offense DUI it's rather to to provide the functionality on the cluster level, the virtualization functionality on a cluster level. Besides that, we also want to play nice with kubernetes, so we want to be well behaving in the community. We want to do that technically, and community.

C

Wise they're technically means that we want to avoid hacks and rather see that we extend communities in a way in which it is intended, and then we obviously strive for production, stable or a stability on all levels. So that means we really need the production, readiness on the low levels so where the VM is really run basically on the bare metal on the node level, but also up to the public API so that you ice or automation can be built against our API.

C

All of that should be looking like a native kubernetes api on the other side to give the user a unified experience regardless, if he's working with containers or pods to achieve these goals, we we had to understand a few things, so we need to understand what virtualization api do we really need we'll get to that in a moment, but there are a couple of options of where that could lie or how it could look. Then we wanted to understand what what run time would do we want.

C

There are also several options: should we use lippert and KVM? Where should we rather drop to qumu, because it also has its benefits and Ronald drawbacks? Then another point is: how do we finally integrate I mean if we know how the API looks and how we want to do the runtime? How do we integrate with kubernetes?

C

There are also several ways, and we will also take a brief look at them and but that also drove our research and then in the end, was really to identify the technical gaps between the virtualization functionality we know and have today, and what communities can deliver, especially in the infrastructure area, so store networking scheduling so prior art. So before we start with Qbert- and there were already quite a few projects without looking at when they emerged here are three I want to speak about? So what is Birdland?

C

That's a project which is effectively a container runtime for kubernetes, which can run VMs, and the intention of the Birdland is really to run diems I mean it's not doing it transparently. It's rather providing some mechanisms like plot annotations to to customize the VM, but on the other hand, it's using the pod API to define the VM.

C

So you take a pot specification and the verted is arriving in DM from you, and you've got some freedom to to specify start and part of it like what kind of bus you want to use for a disk, then there's run B, which is a different approach here, it's not about running VMs, but rather to run VMs for isolation purpose.

C

So what you have here is you also post a VM spec and if it's run by the run, baby run V runtime, then, if M will be spawned and that parts of the containers are launched within that vm, but they, the fact that the VM is running, is hidden from the user in any way and you don't have a way to influence it. The same is true for clear containers, so we both the ICC runtime.

C

There is an implementation which you can indirectly use through the MOC I CRI runtime, and that is also aiming at providing an isolation to pods and not to offer the user the ability to run VMs. So it's also implicit so far, so good. So we've seen that the existing projects over different API is to interact or not interact with IBM, at least and the work that was so far the closest to our approach. But it does not provide the API we need so what we need.

C

They were mainly you so and and run vintage containers were creating dbms for isolation, and the vertol proach has a problem with scales of aura. So if we try to put all the details about BMS into annotations in the pod, that is getting messy and unstructured and eventually does not even scale or we are reinventing a structured struct in the headers of a pod file, and that don't know did not look very well. So we were SCI's said that we wanted to go with a dedicated API for virtualization.

C

We actually have that. Today we were based on TP RS, which are no obsoleted by CR DS. We actually working on our own use, api server for custom resources and sap resources.

C

Because this allows us to define our custom, api's or custom objects or our custom kinds and resources related to virtualization so so far, for example, we've got an API to specify a VM, but we also got an API to to trigger migrations, and here we go with an example of the current VM API. So if you look at it from a submerse and tactical point of view, it looks like a pub specification. It also has a header.

C

It has a kind field to has metadata in a spec a specification, and we can also be written in llamado jason like what specification it's just a different object. You are specifying to give you a little bit of insight of how this translates to to our stack, so part of parts of these specification or parts of these definition are used in different parts and committees.

C

So if you look at the Doudna main section for the below, where you've got the name and the type, this is obviously taken to to define the right VM on the low levels. But if you take go a few lines further up, for example, the note selector, that is, a field which is taken to to be passed to kubernetes, to leverage the community scheduler to do the appropriate scheduling and right at the bottom.

C

You also see- or you get a hint now- that we try to reuse existing API objects and become of the kubernetes api in that example below is that we are using a persistent volume claim as a source or the backing file for for a disk.

C

So that is that was about the API, and you saw that we are using liberabit to run this stuff. We had it at one or two times, but there's more to it. So libera does not run on the node level, which is done by the verb that run being care containers, so they're not necessarily running lipfird, but at least they are run not within pods or containers, but rather alongside the cubelet as a service usually or the host operating system.

C

So what we do is we said we want to container us our runtime so first before we get to that livered yeah, we, the decision when we delivered, because it is a proven, stable and feature-rich component, which were also using in other virtualization products or components and the open source landscape, and we put leopard and QE mu, so all of its dependencies and things necessary to run the games into containers.

C

That has benefits the main benefits are that you can let the lifecycle of livered and all the dependencies are not tied to the cluster or to the node life cycles, and that gives benefits when we think about releasing and updating strategies.

C

Furthermore, you don't need to have two approaches to, on the one hand, many to the cluster or the application within the cluster, so Cubert and, on the other side, make sure that the right dependencies are installed on the hosts, because then you have yeah two mechanism, two processes which you need to take care of to perform, one update which can get complicated technically.

C

However, what we do is that we, if you spawn a VM and will be moved and put into the resource group of a pod, so not audience- are running lis in one resource group, but rather are are moved to resource groups, but we get that into detail in a few slides all right, so we spoke about the runtime and that it's living in pods.

C

Now the question is and has said that the that other approaches put their runtimes alongside the cubelet. So we also considered that approach. But for now we said, that's not so nice because we would need to put stuff on the host. But that's not the only integration point.

C

We've got additional integration points so, for example, we said that we have except we need a separate API for virtualization functionality and I mentioned already that we we were using Co decent TP RS in the past and that's actually the the intended and blessed way of integrating with kubernetes on the API level, confusing either CR these today or custom API servers.

C

But the good thing.

A

C

Is that if you provide, if we do the integration on the API level, on the one hand, we can use all the tooling provided by kubernetes to do that integration? On the other hand, we can actually use other kubernetes objects, for example, I mean in the API example I gave. They will reuse that persistent volume claim inside a VM do to make that semantic connection between WebM and using a persistent bottom claim as a backing toward for virtual disks. So.

C

With all this knowledge, here's now a technical, deep dive and if there's a question which I could imagine, please let me know now, but with these informations we had so far, the architecture looks as follows. So, on the left hand side in the clustering components, if you look at the white box, which is called API server, that is the API server of communities conceptually. We now run alongside of that. Our bird API server today, it's because the API, so variation is really fresh and we are still updating that part.

C

It is a bit different than shown in the diagram, but it's really close to that, then to add our virtualization capabilities to the whole cluster. We use a daemon set to deploy the vert handlers and liberty to every host in the cluster or to selected nodes. So here we can also use the namespace features or node selection features or in general tank scintillations or labels to assign our cuber nodes to to the real physical nodes as needed.

C

The verb handler in the picture, which is shown quite in the middle, is responsible for watching DVM API objects and then performing the necessary operations on the host. So speaking to looper to really launch the VM and SEC it's entered here. The VM is then actually on launched in the poll of the handler and Lippard, but rather the pod, which is assigned to that VM.

C

That is important to do, because it allows livered sorry allows communities to track the resource consumptions of that specific VM, and not all the resource usage of all the MSS aggregated in in one pot, so we're basically doing everything inside pods and that also permits, though we need to see if it's feasible or not to run other pods, which just run container water runs alongside alongside super with that API server aggregation, which we saw on the left side on this side slide. So here you can really know, or that allows us to really integrate.

C

Also on the on the on the user level, with the existing tools and communities. So tube CTL will work for for a pod like it will work for IBM, because the API server will make sure that the requests related to virtualization will be redirected to to our virtualization logic, tour boat controller, which was shown in this slide before so I described it a little bit and now would go into a small example of how it really looks to illustrate what component isn't both where and how does the spawning process of a VM look.

C

So this slide shows howdy of the waiting status, so nothing happened, no VMs are defined and we want to report once the VM is getting created. Currently, that is a CRD. The vertical troller will see that actually, because the vert controllers budget, watching for VM instances or actually all kinds of virtualization objects which we introduced once that happens, the vert controller will schedule a part in the so-called VM pond. This part will act as the harbor on the resource group for that VM.

C

Once that pod is scheduled on a VM on a node, the bird handler will see that the VM work scheduled it will tell libvirt to launch the VM inside that pond from that moment, on. Kubernetes is aware of the resource consumption of the VM and can do proper accounting. So we can even set limits on the part. I mean limits in the pot sense with regards to memory, consumption and CPU limits to make sure that the e/m really does not exceed those limits.

C

This cannot be done for one VM, but rather for all the ends, so the trick that the process here is pretty simple and straightforward I'm. Just looking at the questions to see, if we can answer something.

C

So the question is: can the note in this case be by metal or virtual or on your one of them? So it can be both, but it depends so primarily cubers intended for for my metal, that's what we traditionally run virtualization workloads, but nesting is becoming more more stable, I'm, not sure if we will support that in production mode. But if you enable nesting on your physical node, then you can also run Cubert on virtual nodes. Oh and besides of that, if you don't enable nests, you can still run your VM in emulation mode.

C

That has a serious performance impact, but it's still possible. The other question: oh yeah, right there wasn't answer very good.

C

So that is effectively the launch process. That's all happening in the kubernetes cluster. There are no underlying components involved. So far there are micron. There are some things to consider on the low levels which you cannot see here so which empower bit nasty and to be solved, but the proof of concept works and we'll take a look at that, and not only the proof of concept, but it's working reliably. So far. We still have to do some research.

C

That is rather the approach here so technical gaps, but in general we saw while, while starting or since we started working in Qbert, we saw that a lot of functionality is there. Sadly, as often it's the case that the functionality is present in areas which we don't need that feature-rich yet and that areas which we require is lacking functionality.

C

However, wherever we see gaps, the main intention of Qbert is not to work around them in our on our side, but rather to the kubernetes community to see that we can fix those issues in communities if it makes sense and if it's possible, why?

C

Because we wanted to trench them and we want to improve the kubernetes infrastructure like if we take core concepts like anything, we could possibly work around them in on our side on Qbert, but it makes so much more sense put to put that into communities so that containers gain these benefits as well. So that is that is one of the conceptual also reasons for kuberan that we want to drench from the infrastructure of kubernetes to also support our workload.

C

If you address in the details, then we can take a quick look at all the gaps we encountered or at a selection of gaps we encountered so far, so resource protection, exclusiveness of fencing, ET momentum, fencing exclusiveness, is really that you can guarantee that only a single node or a single VM is accessing a volume at a given point in time. Currently, this is done on D on the scheduling level in communities, but that's not sufficient. If you want to prevent that, the database is getting corrupted host.

C

Lifecycle management, which is strong at least, and all word name- is also something which is not so good. Yet in kubernetes, cutemon and tools alike really improve the situation, but we're still seeing some gaps. There device management, it's seeing some traction, especially across from the vgpu point of view.

C

So there are some concepts pushed into communities to support device, management and device path through inter pods and thus, hopefully also into VMs compute for performance. We need to have CP, opening and Numa we're also seeing some advancement there, and it took quite a while for communities to find the right approach here, but it's really nice to see that they warmed up to it. Dynamic SLA is effectively that we can modify the CPU and RAM limits in the pod sense or that we can remove and add RAM and CPU for VMs.

C

That seems also to be happening, but it also takes its while until the concepts already in communities and once they are then can feedback that I'm just checking the questions.

C

So in this sense of multi kinetics clusters with Federation- yes, we need to see how that works. I'm, currently not aware how that works with the custom API servers in general, but we are lining to to what communities expects, how how integration should look, and so we hope that kubernetes also provides a way hopper of how this can look in the in the broader communities ecosystem, including figure Federation.

C

Health networking handle between contains and VMs within the same project right. So that's actually, oh what a coincidence. So it's actually the next point of the detail gap so currently or work in progress, and it will heal and within the next two weeks, probably is that we are first now it took us a while to find the right approach. So our first approach will be that we obey the networking of kubernetes. So what we do is we request new interfaces from kubernetes to be assigned to a pod.

C

This is actually in line with what is happening in kubernetes, so pods will get woody network or multi interface support, and we will be reusing that so additional interfaces will be used first to be assigned to VMs. That is pretty straightforward, but then we need to make sure to align with kubernetes that we give that the VMS are respecting the IP addresses. We hope that this will work out. We've see an ISO we're doing that on the CNI level and we effectively request new interfaces from CNI to be attached to VMs.

C

This is under heavy research and we are prototyping. This kind of stuff we are in contact with the CNI over several CNI people. There see what the right approach is on the long run. However, we want to see that we get layer, two networking, but that is conceptually difficult, because later networking abilities is not there yet, and it has not been discussed so far and the question really is if it makes sense for communities.

C

So it's a research, research area which we need to tackle for storage. We identified that the cubelet does well at setting up storage but for advanced production, ready configurations of multipathing, for example. That's still a way to go there to support that for every connection for quadri, while you type it support advanced operations by cloning, a snapshots, especially on the on the server side, is also something snapshotting has been designed. Cloning I've not seen any proposal for that yet, but that is stuff.

C

We need for DBMS scheduling, resource driven, so this is actually also emerging since we created that slide deck resource aware scheduling will make it into communities. The first proposals are in rewrite related to device plug-ins, but we also need to do the scheduling based on custom metrics.

C

So so far, kubernetes is using the pod metrics to do the appropriate scheduling we need to it, enhance kubernetes or provide our feedback to make sure that we can also use virtualization metrics to do the scheduling rescheduling of balancing is to make sure that the that pods today, but the only scheduled once they are created, but to apply policies to a cluster, for example, for power saving, as et mo mentioned before you want to do rescheduling or band balancing.

C

So after pod creation time, you might want to move a pod or a VM, in our case, to a different node to be able to shut down some notes to save power, I'm skipping the modularity on the infrastructure side. We also obviously need some. We need some enhancements to make it really to allow us to extend communities to provide the virtualization workload.

C

So we want to extend cubes ETL to to be able to add verbs like live migration or starting and stopping off VMs, which is not present in the pod concepts, and we need the en add-on formalization to make sure that we really are on the same page when it comes to integration between communities and and and couponing, and then there's the UAS native object, storage in kubernetes, which effectively allows us to to really now that we have our use api server to to store our objects still in kubernetes and not need to provide our own key value store to store our or data.

C

This got very detailed, I noticed, but I hope it gives an insight into what gaps there are and what we are looking into to solve. So a quick wrap-up before we get to a small demo. So on the API level, we really we're investigating. So we start with a livered like API that I'm, including scheduling aspects, and but we still want to see and what I get feeling for what granularity do we want do we want to be more polite, which was very simple at the beginning.

C

Actually innocent has now grown more sophisticated with all the enhancements like, for example, advanced selectors, taints integration, it's adding more complexity to pot specs, but we want to see if we want to say complex or become more simple.

C

We also need to rethink on a resync but continuously think and revisit if how the low-level mechanics, what I mentioned initially doing. The implementation on the CRI level has benefits and drawbacks, and we want to keep revisiting that process, because we need to evaluate that for every single feature we have and how it matches together its ongoing research and often you just get to the questions one year once you get to the implementation.

C

We notice that community still has has the gaps. We just mentioned them, and we need to see that we we get the right ones delivered in time, so that we can make progress and and are not blocked, because otherwise we are pushed to do workarounds and that is is taking time and doesn't drive communities in the right direction and I cite note is that the operator pattern world. It works pretty nice. That is the pattern we used to track. The VM Tsin was actually defined by core OS.

C

In the end, we want to say that we need to continue with our research.

C

We see that that there is potential for convergence and it looks promising like one nice side effect, which is implicit, is that it's so convenient that you just take a cluster or like we will see in a minute you take mini cube and just have to deploy pots on it and you get a VM runtime for free and you can use the same infrastructure that is really really convenient.

C

These separations are far also looks like it win win because kubernetes can drive in its own pace, focused on the container workloads, and we on the Qbert side can look, and then we drive an hour pace and and just see that we aligned on the on the integration points in general. It also looks like a win-win because we can, which goes into the direction of the previous point, because we can align on the same infrastructure.

C

There are tension, points. What we need to see what we need to to see. What role do we take? Do we drive for the absolute feature, parity and and even violate some kubernetes assumptions, or do we rather say we limit that feature, but gain that we are more behaving like kubernetes would expect it or how communities is is setting expectations like for put 1 only 1.

C

We know we need to do a lot of detail, work to get it the real performance out of it and we're not so sure if communities will go the whole way. So that is an area where we need to see. Where do we end up with, or what do we end up with? What? Where do we want to put the focus on full functionality or better integration, and finally, we're seeing a lot of stuff is easy, but really coming with stable and production.

C

Enterprise ready the solutions is hard, and we see that every day after these, after this small summary, you know it's close to the end, I would still try to give you a small demo to show how this is actually working and after the slide has been shared. You can also try it yourself at home. The demo which we've prepared is based on mini, cube or is Larry leveraging mini cube, and it will effectively deploy to below mini cube and we'll be able to run a VM and connect to it.

C

Hopefully, before that I'm telling you look at the questions, what they seem for the questions, I think.

A

You're doing a pretty good job so far answering the questions in line so yeah. Let's, let's see the demo and then we'll get more questions after the.

C

Very well so here I've checked out the demo repository, which is shown in the slides.

C

To start you, you run the demo dot sh script, which is effectively just checking if you've got cube, CTL installed if you've got mini, cube installed, and then it's checking out q board from github after it checked out q bird, then it's deploying manifest, including a test vm once that happened- and you know deploying manifest is one thing, but then it takes a while until all the container images are downloaded- and that is what is happening in this line now we're waiting that the containers are downloaded and if you do it for the first time really it can take a few minutes until all images are downloaded, there's a lot of debugging stuff at it.

C

It's not optimized for size yet so it can really take some time at the moment, but once that is done, you can really see that iBM is eventually running and yes, we do see it here. So DVM is running and here by the way you see, the output is not very sophisticated. So that is what we want to improve. Cubes ETL to allow custom rendering for custom objects VMs in our case, to get some more details, we can do use the JSON output, and here we see already the VM definition in the domain part.

C

We see that a console is defined in a disk that a nice gauzy target is used as a backing source for for the disk, and this should actually be updated to use a persistent volume claim toilet that more and we see that a graphics device is attached, including a network device further over above just to show that it all really looks like like a regular. The regular kubernetes object. You got labels, we've got some metadata annotations, that's really something you can use with your existing tools.

C

To see so, we know that the VM is running. What we can do now is we can take a look at the pods and here and it should roughly match with what you've seen in the architecture diagram so because the H I proxy, which is just an artifact, because we don't have a PS of aggregation. Yet we've got nice scuzzy, demo Targa, which is providing the image for the VM. We've got limbered running a dedicated pod. We've got a spice proxy, which is sitting on the border of the cluster to allow accessing the VM.

C

We've got the bird API, which is providing the custom resources, or that currently is providing the validation of the customer resources. We've got a bird controller which is reacting to to the created API objects OPM. In this case we've got a bird handler which is running on the node and speaking to Liberty, and we've got the bird launcher. That is the pod VM, which is opening the resource group for the VM, and we've got a vert mighty fest, which is an artifact of the work-in-progress we currently have. If we now do.

C

If we now look into the Lippert pod, we should be able to see that the chest VM is really running, but that's not what we want. We don't want to directly go to the pod, but we run, and we don't want to go to the port. To then see that the VM is running. What we rather want to do is we want to connect to the VM itself a puzzle.

C

So what we can do this there's a small binary utility in which we provide it will become a cube, CTL plugin at some point once that is standardized and looks it is now verts. Etl allows us to connect to the console and actually also to a spice. So what we will do is we want to connect to the serial console. First, we need to provide the the right IP to connect to of the API server and it's listening on a specific port.

C

Then we need to name the DBM and the device to use all right.

C

Just a second all right, probably use the LAN port. I am Nancy, I had a mix up the port number. So- and here we go, it seems we logged in.

C

So here we are running in Alpine Linux, and that is actually the disk which is provided by the demo target. We can do stuff like rebooting, so it's really a connection through both CTL, which is actually in the background. The the implementation is similar to the exit call of the cube CTL command. So here we really try to play along with how how communities is doing it.

C

The serial console is not the only thing you can also do is we can do spice. We take the same line.

C

C

Oh yeah I should shut it down. So what we do now is we keep see deal.

C

So I delete it. The chess p.m. and now I'm creating it part time for demo web cluster, llamo IBM is created and now should be able to anticipate the spice ball, and here we go and we see that the VM is running. It's a bit slow because the question was this is running in emulation mode.

C

We can actually change that up. To recently mean EQ did not allow during nesting I, think it landed in the last version of MIDI cube, so it should be faster in future. So, and that was the demo any questions. What is showing that I can not show like migration, because it's just a single node, but it's sufficient to to play around with other stuff.

A

There aren't any questions. I have maybe it's a newbie question. The the issues you were talking about with Numa the memory access bottlenecks and stuff is: is that anything that yeah I know that's important to all the the VM and virtual machines folks out there, but is that anything that gets solved by kubernetes that bottlenecks I know it's? You keep talking about it being very, very important to solve that.

C

Wandering so yeah so sort of interrupting the delay is horrible, so openshift is taking a lead here. So what community says- and that's also by the way- an argument for putting VM sin in VMs into pods is kubernetes says we want to support any workload and there's that area of performance sensitive workloads and which OpenShift is seeing a demand for, and so some people from from the openshift team, like jeremy de, is looking into that area and they actually tried for three quarters of a year to make.

A

C

Aware of these important tuning on the low levels, the problem is that in the community, stakeholders said for a long time, that is too fine granular and we we cannot solve that on the API level, big, because we don't want to to mess up the API with all those low level details. So, after an iterative process of like six or seven PRS or proposals of how to solve the problem, they finally settled down and found something. So today it looks like they will be implementing an approach where you can.

C

We can specify policy of how your processes will be distributed along Numa nodes and which is a compromise so back to a question. Yes, we do see that they're opening up to it. It just takes time, and especially in areas which have not been important to communities before it really takes its time until the use case is understood and until the right solution is found, it really can take a little bit of back and forth.

A

So if people want to get involved in this project, where we're on github, can they find you? Where can they find you and do them there? We go that's to see.

C

Yeah, thank you very much. So first thank all of you for listening and, yes, we are really looking for for people trying out coopered and trying it and finding issues also from another distribution. So far we are trying it on CentOS and fedora, but we would really like to see that it runs on other distributions like Debian cores as well. There shouldn't not be it shouldn't, be so difficult because we are or tag our platform is the cluster and not the host.

C

So it should be fairly easy, but that is what we really would love to see. Contributions, but you're also welcome to contribute to code, doing testing or just use it so and join us and get up and cue bird. We don't have a fleck channel yet, but we on freenode as well on our I, see good old IRC in the Qbert channel on the freenode network. So.

A

There's one more question that marchello is asking is: how can I look the logs between look at the logs between the VM, orchestration and kubernetes cluster in a kubernetes face? They have fluent epod to catch all the lot? What do you believe people of all bosses, the basic quotes.

C

So that's a good question, so what you do is so we try to integrate nicely here, so we provide the logs from the pods. So if you are interested in what the controller did to schedule the node, then you can really take a look I'm doing it now you can take a look like with any other component, like with any other pod at the logs of the relevant component.

C

You want to you want to debug, so we are emitting event. Actually, we are meeting vm events on the cluster for state changes and we try to emit valuable logs for every component, so you can debug it on the note level as well, so you can say pods to get the right.

C

So if you want to debug the note level for a specific note, because you know that a specific note at issues, then you can take the handler and take a look at them there, and the same is true for a poor livered that you get all the livered given output. So it's fairly straightforward.

A

I think my mind is a little bit blown because it's like I, always with kubernetes, as everyone knows that we always associated everything with containers- and this is you know, a huge leap forward and I've talked with the clear container spokes a bit too in the past, and this is just you know the merging of two great technologies in one one, cluster management approach. So I think this is just pretty useful stuff. Is this?

A

um It yeah pardon I'm at Red, Hat and I'm, asking this question and it's an external facing thing, but is this mostly Red Hatters working on Qbert at the moment, or are there other folks from the kubernetes world walking music guys.

C

So we emerge from redhead because we did not see that that these needs were addressed elsewhere. What we see now is so first we also, we see participation from OpenStack, which is nice, so we're classically from from the overt side, but OpenStack also has it interest in it, and we also see that users come from different distribution. So we especially see that bugs are filed by users from the outside, but the development, as so far is yeah. It's a reddit, but I would really love to see that we see more contributions from from others.

A

One one of the other things the on the kubernetes community meetings. Have you done one of their um like the the intro of every I? Think it starts in three minutes? Is the next one? They have little demo times where you can demo your new stuff? Have you done one of those? Yet? Oh.

C

We are we want to the demo. I just showed is really hot. It was effective, it's just possible to do that since last week laughs, so it's yeah there wasn't to demo it before was really cumbersome so doing the demo with mini to bring easily forward for us, because we can now easily show it and that's actually the opportunity that we want to step up to. You know other forums to to show what you can do all.

A

Right, well, let's get you on there because that'll get you a lot of eyeballs from around the kubernetes world too. So this is great work. Thank you admire and Fabian for doing this today. You know from the audience it's a really good question. So thank you for coming and asking them and we'll have you on again in the next iteration too, and hopefully we can get you to show it with mini shipped or maybe even open ship live at some point.

A

That would be awesome, I'm not to put you on the spot, but that would be really really cool, and this is you know it's very interesting to me, because I think all this mixed workloads is definitely something people are looking for, as it's obvious from the audience that we've gathered here today. So hopefully you can get you some feedback and more more eyeballs and more resources working with you on the project. So, thanks again for coming and we'll talk to you all soon,.