Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Minikube - Bálint Pató & Thomas Strömberg, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive: Minikube - Bálint Pató & Thomas Strömberg, Google

This session is for new and seasoned minikube contributors. If you want to develop a deep understanding of the minikube architecture and see how you can contribute, where the product is evolving, explore new features and have a discussion about the future of minikube, this session is for you.

To learn more: https://sched.co/ITDI

A

Good afternoon, Q Khan welcome to what might be your last talk today. I assume.

B

So I hope so there's no other told yeah. There's.

A

Always the hallway track.

B

A

Welcome welcome to mini cube status where we go a little bit further than we went with the mini cube start talk earlier.

B

This week the intersession.

A

All right, a handful of you excellent, shall we start yeah.

B

I guess we will go through some of the initial things for those who weren't in the intersession. So, for example, who we are my name is Balon Tatsu I work at Google for the last year and a half now and and started to work in the container tools team like three four months ago and I'm, leading the effort around mini, cube and scaffold so and help out with Kenny ku as well time to Tim and.

A

My name is Tomas Stromberg and I to work for Google about three months ago. They offered to pay me to work on many cube full-time. So that's what I do now and I also worked a little bit on Curtis.

B

A

So agenda today is a quick architectural recap for those of you who did not attend the intro, a couple of feature: demos for new things, a little talk about tinkering with mini cube, depending on timing plans for 2019, and we will leave time for questions, feel free to ask questions during or at the end, yeah.

B

We do want to have this one much more interactive than than the intro was so feel free to jump in ask questions. We go as deep as we can in this 35 minutes and yeah yeah. This is more.

A

Of a shallow dive than a deep dive given the time constraints, but we'll do what we can so. First, we've had a lot of contributors over the years. I think it was 387 by the last count.

B

A

Jesus all right well, thank you. Thank you to the 13 or I drowned. Thank you through all the people who who have contributed, especially the new ones, yeah. We couldn't do without you, so.

B

A

A quick architectural recap, so what is mini cube? Well, I, guess how many of you have not used mini cube before all right. So a couple so when you run mini coop start it's effectively going through these different stages of setup. We've got a machine driver that sets up a VM provisions. The machine in this case it's KVM, but we also support a huge variety of other things. For instance, VirtualBox VMware, Fusion hyper-v hyper kid.

A

So we provision a machine and what's the first thing we do when we get an on to machine. Well, nowadays we run cube ATM to set up kubernetes and we set Cooper at an ease up with real live released, binaries and it's a real normal released, cube ATM. The idea with mini cube, is once you get the VM provision and cube. Atm is done. It's real kubernetes. It's there's not really any magic involved. There used to be a lot of hacky things and before cube, ATM was involved, but that's it for there.

A

So we also run a container run time by default. I'd stalker here, I selected cryo, we're gonna, actually do a little demo with container D container D and G visor. On top of that, we run kubernetes. The default right now is V 110. That is so much shameful to me, because I really tried to make V 1 2 up the default and I failed because it was actually it was too slow to provision.

A

So we backed it out as the default, but we hope to do I'd like to do V 113 by the end of the year. Now that I think we got the performance kinks out, so we also have a vast add-ons marketplace as you as we'll talk about a little bit. So here you know deploying ingress, and then we have your app, which is the whole reason you started on this long journey so so new feature demos, alright,.

B

So I wanted to show a very similar demo that I did in the intro, which was running a couple of things on mini cubes, so mini cube currently is up keep status. You see that it's running I started this mini cube instance with a little bit more complicated command. So let me show that yeah so.

A

We so the reason why we're starting these before the demo is because mini cube does not work very well with port connectivity and we've had full of poor connectivity here. Unfortunately, just didn't want to.

B

A

Yes, we would no risk here, but.

B

So this is the start command that I used I used, VM driver, hyper kit I'm using a little bit larger disk size and the container runtime is container D. So that's what is running now in the background and I'm going to start mini cube tunnel. So this is a new command that came in with the last release. 0:30 1.0, I need to type in my password because mini cube tunnel actually creates network routes into the the service cider.

B

So we can actually start to directly address services that are that are running on the on the cluster and also a mini cube tunnel. Has a load balancer emulator. So if you you, you run something: let's deploy something so cube: CTL iran and genomics image engine acts. So this will deploy a pod. You see that here, I have the cube. Cito get pod from all namespaces, and here I have services.

B

Is it legible everywhere in the room, large enough phones, cool, so nginx spun up, I'm gonna expose this now keep CTL expose deployment, nginx type load balancer, and this is what previously was not really working in mini cube. You needed a cloud provider to to provision a load, balancer and then inject the ingress IP into the the Status part of the service. So, as you can see, mini cube tunnel recognize that nginx is running and picked it up, and then it copies the the cluster IP into the external IP.

B

So now you can use the same scripts use you would use in in in production to to get the external IP of of your your services and then you can actually navigate to it and and it should work there's there. It is welcome to nginx it's not yeah I.

A

Was gonna say so to recap: if you, if you're nuts, specifying type equals load balancer, you don't need to run mini cube tunnel, but one of the whole design goals. Around mini cube is high-fidelity simulations of your production environment and if your production environments using load balancers well we're not really load balancing, but at least your scripts will still work.

B

Excellent another thing that I set up now is a a little bit more involved, but I have.

B

Dns setup, which is pointing to 10960 10, which is cube, DNS, is IP address, so my Mac host is going to resolve everything under the SVC cluster local sub domain through this name server. So, theoretically, I should be able to do reach this nginx server through nginx, the default SVC cluster local, and there is so excellent why this will be cool I'll show that when we are actually deploying another application using docker scaffold, another thing I wanted to show you what's running as add-ons so mini cube.

B

Add-Ons list is high list add-ons and then currently we have the dashboard, enabled the add-on manager itself enabled as an add-on. We have a registry enabled, and we have G visor enabled. So we can also see that registry is running here as a the service and as a as a pod. We have the G visor pod running here, making sure that the configuration is the right one, and so let's try this out. So what is G visor, so G visor is a Google developed, sandboxing technology. That is, basically, you can think about it.

B

As a user space implementation of system calls or kernel. So it creates a much more secure, sandboxing technology for for untrusted workloads. So if you have something like a multi-tenant setup or or workloads that you don't, you can't really trust, then you can just annotate them with with an annotation like this and then they're gonna be running in the G visor sandbox, which is run SC. So, let's see how this works so if I say cube, CTL create you have this untrusted yamo I, just basically just another engine X then should be created.

B

The pod is created. Let's see if it's running it is a spun up and then one way to test whether you're running in G visor is.

B

Exiting inside this pod and checking the the syslog, so the reason why this is one way to do this, one other way could be to just PS saying within the mini cubed VM, and then checking whether it's wanna see the the process that is running it. But even that is not like the rawness to binary could be something else as well. So.

C

B

The recommended ways to run this now there is a bug currently which will throw an error message so.

A

B

One this is this yeah.

A

This add-on was pushed on Monday yeah, so but there.

B

We got it. This condition second run will fill 6.

A

B

And then you will see that it was actually starting G, visor and, and then this is this is what you want to see when, when you have G visor a disaster, so if I run something similar on a non-jew visor workload, then it's it's a different regular run, see syslog.

B

So that's a minute demo off of the G visor add-on, which you can just simply just turn on with mini cube, add-ons enable gy0 and.

A

Similarly, if, if you have add-ons, if you have a product where you think people will really use it in mini cube- or you know, random service that you would like to see tested and it's complicated to set up, the add-ons is basically a marketplace so feel free to send. Pr is to add your thing to the add-ons right now we don't have the best model as far as updating it between releases, but we hope to release often enough that it's not painful so that will change.

B

So another thing so because registry is running now as a as an add-on, it is exposed as a service. I should be able to resolve this DNS name as well. So if I curl I should just get an empty response, it's it. So one cool thing about this is that I can use now this registry from my local docker client, to build images and push push them to this registry, which then container D itself can use to pull from so there I'm cheating a little bit there is this this something.

B

This is something that I opened an issue this week about that we should have in the container the config. We should have the insecure registry set up for for the internal registry, and then this whole thing will work now it's set up currently so so it will work. I just wanted to set the expectations. If you try at home tonight, this might not fully work yet all right. So let's do another test.

B

So scaffold is another tool for local developer experience that we are developing and it helps creating a continuous development workflow. So as as you change, your files, scaffold will rebuild your docker images, tag them test them push them to the registry and deployed the the manifests that you specify in the scaffold llamó.

B

It will deploy them to the current cube city. Our context designated cluster so currently in this project, I have two services, two artifacts that are being built, one is for listing contributors from the mini, cube, github project and then a front-end which will generate that nice slide that you saw about all the contributors of mini cubes. So these these are here as two sub projects. This is a little go project that just builds the go program and then a front-end is a simple react.

B

Gs project I'm, not a front-end engineer, so it's gonna be very simplistic. The code for this is in in github.com / baba, bhot / country wall. So you can try this out. So let's try to run this safe scaffold. Dev will just run this and and and try to do the the deployment and the pushing. Now you can see that the images are defined, as, as my repository they're, pointing to my my registry and in GCR, so I'm gonna, repoint scaffold to use the the local registry inside mini cube as a as a registry.

B

So that's gonna be the default repository, so it's gonna rewire these artifact image names on top of this. So let's see how we deal with this. Another thing is that there's this warning, so this old switch also shows that we haven't really tested scaffold yet with where the container the mini cube set up. So scaffold assumes that when you spin up mini cube, then there's gonna be a darker demon inside of it, which you can reuse.

B

If container D is enabled, then the darker demon is not going to be running, so it is falling back to to my local darker demon, which is running here and by the way that darker demon also has this registry set up as an insecure registry just to have one more piece of the puzzle and let's see what happens so dark darker build was done. It pushed most of the layers were already existent and now we're building the second service. The front end service, which is a bit larger.

B

So it takes a little bit of time to descend the build context to the darker daemon. But after that it should be relatively fast or build it push it and then the kubernetes manifests will be the the the next step deploy. So in case of the front end, it's a service that exposes its of type load balancer, so we'll we should be able to see that as it pops up and the backend deployment part is as just a simple pod with a replica of one.

B

The backend is very similar. We have a service named as we see, contributors listens on port 8000 and you can see that I've prepared I'm going to run this in G visor soon and then just a simple pod, with a single container just have to understand how the two services talk to each other, this guy, and so the front end, is talking to the back end through this proxy.

B

So it's referring to your SVC contributor service by its its DNS name. So that type talks, that's how these guys talk within the cluster to each other excellent. So now the second push was done and then the two services and and and pods are created and let's see what's happening so here we are. We see that we have the the front-end service and we have the SVC contributor service.

B

We have the the pods as well deployed and happily running so now, if I do front end the as default SVC slash cluster local, we should be able to see the wall of contributors for mini cube, so this is coming. This is how fast the Wi-Fi is and yeah. So this is great. So a couple of fancy thing that scaffold does to make development easier. As, for example, you can define some of the files that scaffold is watching automatically based on inspecting your docker file, but some of them.

B

So if you choose change files in your directory, it will we trigger the whole darker, build retag re push and redeploy, but you can mark some of them as as sinkable files, and then these files are not going to trigger a full rebuild they're. Just going to be singing into the pod and the running pod directly, so let's, let's do that and and we'll see a quick change. So if I say hello, cube, Conn last presentation, awesome so I hit save and it sync that so it's already there.

B

So if I alt tab, you can see that it's already updated. So that's pretty awesome and then let's do just another test quickly. So if I change a file that is defined as a kubernetes manifest it's not going to trigger a build either, it is only going to trigger the redeployment. So let's try that, with the with the service contributors, back-end I'm gonna turn on the the annotation 4G visor yeah and.

A

This should generally just work the same way unless you're using a fancy. Sis call that G visor doesn't support yet, but that seems unlikely, especially for a way that.

B

So it started up and the logs are being pulled and it shows that everything is healthy. So if I just refresh it's still working ish, okay and then now we can do our test Witter.

B

Really we are running in G, visor or not. So if we are gonna try this guy- and this is the pod X akin to it from the message. Okay, this is expected another expected and there it is so it is. It is running into you visor now, so why do we do this? Why do we? Why did we put you Iser into into mini cube?

B

So the idea is that G visor is going to be of GK I. Think the GK alpha it's an awful feature in GK. Currently so sooner or later, this will be a thing that you want to run in your GK clusters. So it is very beneficial to be able to try that out in your local environment and test it first and not get surprised in energy k+ surfer. So that's the idea so try it out play with it.

B

Provide feedback on on the G, riser project or Dominica project depends on where it breaks and yeah help us improve it. So.

A

So many cube aims for very cheap experimentation so that you don't have to pay us money if it fails. We'd like you to pay us money, anyways yeah we'd like to get money for a success, not failure, so so X.

B

Sorry I'm done with the demos we still have time, yeah.

A

So I wanted to talk a little bit about plans that were plans for 2019 and with with the caveat this, this is all fungible I mean this is based on the community, so the plans are basically whatever we and what we vote on as a community or as developers. So these these were. We haven't actually like written up our roadmap externally, yet for 2019 we haven't, but we've been starting the discussion this last week and here's a couple of things that we have come up with and talking with people.

A

So one of my personal pet peeves is that mini cube is too damn hard for too damn many people. There are a lot of situations where it breaks I mean if you're already a kubernetes developer. You know you can probably figure your way out through it, but for a lot of people.

A

Actually, roughly 20% of people based on the survey we did mini cube is how they learn kubernetes and if your first experience with kubernetes is a bad one. Well, that's, you know really terrible for the community, so we would like to do a single step installer, which means you know the idea of bundling VM drivers or Auto, selecting which VM to use. You know if you've already got a machine with VMware fusion, we shouldn't, you know we should just mini cube, start should just work without any kind of options.

A

Basically we'd like clear end, user documentation and help is definitely wanted here right now. The mini cube website is very code centric and it's not very user centric and some of our documentation is on one kubernetes I/o website and some is on our github and it's a bad situation. We'd also like to stop asking people to delete the mini cube folder when things go wrong. It's definitely a facepalm moment we'd like to make it easy to debug.

B

So one of the things is that so mini cube is not that complicated, seemingly because it it's just a VM plus cube admin. However, there is a lot of things that can break so and a lot of times. People just see that things are hanging or slow or not working, so it is, it is actually hard to to pinpoint, and it requires a lot of back-and-forth on the github issues as well to nail them what the actual issue is, so we want.

B

We really won't think that it would be very beneficial to make this make this whole experience better, to make it easier to debug to to pinpoint where the issue is have better health checks, better preflight checks, if I made a mistake and I have to wait, 15 minutes to to learn that I made a mistake and that's why it's not running and have to go through again a 15 minute thing to try it out. That's a bad feedback loop. We want to make that faster, clear, actionable error messages and just better debugging experience in general.

A

And we would also really like to be much more inclusive. Who you see on stage is not necessarily who the users are. We we don't represent all of our users. We, you know we're a handful of maintained errs and, unfortunately, a handful of maintained errs pigeon-holed into countries that have really great connectivity, and not all of our users do and so I think mini cube suffers for those users.

A

So this one of the things I called out here is I'd, really, like mini cube, to be offline friendly. If it was offline friendly, we would have shown mini cube start at the beginning of this presentation, and help is definitely wanted there and.

B

Yeah efficient open-source governance I put this down because I think that mini cube was a little bit of really grassroots, just quickly get something done in mid, 2016 type of project and- and there is not much thought put into yet like how to make this a self-sustaining project in terms of the community. So this is new for us, though, so we really want help and and from the community. We will ask it on the C cluster lifecycle, meetup group group, as well on the meetings to help us in defining. How are we prioritizing things?

B

How are we going to involve other people? How is it going to be a really self-sustaining project? That is a we believe is a key piece in the in the whole ecosystem to be a free, open, local experience for kubernetes. So that's another thing that in 2019, I would definitely like to to see to have a larger set of maintained. Errs will have a clearer process in terms of communicating priorities and putting this roadmap together. So it's not just us making up stuff yeah.

A

So a couple more plans as I stand: freezing under this blower, that's over my head, I'm gonna, so I. We we definitely plan to continue our investment in CI I've spent, actually, most of my time as a mini cube, developer. Fighting the CI I would like to stop doing that part of it is we're migrating to prowl in the coming quarter. We've spent a lot of time developing tooling, around CI, managing our flaky tests and we'll continue to do that.

A

One thing I would also like to see is more performance. Work happening. I would like many cute provisioning to be drastically faster than it is, and I would also like it. If my MacBook Pro didn't burn me in the lap when I was using mini cube, so there's a couple ideas here. Help is definitely wanted on all of these items and there's been some discussion this week or ad multi-node. We avoided it for a long time because it wasn't really in our original charter.

A

People weren't really thinking about it, but the kubernetes is a lot, has a lot more going on now than it did two years ago and like h.a workloads and things like that, where people would want to have multiple nodes to seem to run their simulations. So PRS are definitely welcome on that and in general you know, help is wanted. So, let's build something beautiful together: it's not just code that we need help with. We really need help with documentation.

A

Project management, I I am not good at that, so so yeah. So if you want to get started, you can look for issues. We've been tagging some of the issues with Help Wanted and good first issue, and sometimes both we just created a developer's mailing list, mini cube. Dev, don't be surprised if you go there and see no emails because we literally just created it there's also an analogous users group. We are pretty active on the mini cube channel on slack.

A

We also have an hour office hours. Bi-Weekly I, put a link there to our office hours, notes that we take.

B

It's 9 a.m. and 5 p.m. PST, so we don't meet twice the same day. It's right.

A

Right, it's alternating.

B

A

So a lot of our developers a lot or you know, especially our developers, who work on mini shift, which is a fork of mini cube, are in Asia. So we try to run in both time zones to be friendly for everybody. We have a survey here and we would love to get more feedback. It's a five question survey, so it's not gonna take a lot of time.

A

We basically want to know why people are using mini cube and what we could be doing better and then Twitter handle is, if you just want to harass us after this, so so questions now I will walk around with the mic. Anyone anyone have questions.

B

D

So in so when you're working on add-ons, how difficult is it to create add-ons on this work, especially like I'm, looking at adding thinking about adding cada um add-on for it? But what would be the kind of work involved to have that happen? Because cada comes with.

B

An add-on is not that hard. There is actually a a description as well how to add an add-on.

B

There's an MD here somewhere, so I think there's I think Annette done there. It is okay. There might be a better view for this. So there's a description here, but it should be relatively. Let me just show you an add-on so, for example, the registry yeah at.

A

Don I, like G Pfizer, actually might be the most relevant in this one. Freaking shot it's complicated though, but.

D

A

Yeah that and that's where things get a little strange so.

B

With G Weiser, what we do is so this pod is gets deployed, and then this pot is, is it actually has the code here as well, and then we have the the G Weiser code here disable and enable- and these are the things that get executed. So basically, this does all the setup downloads, the necessary binaries for for the container DM run. Sc sets it up and then so this this is the pod that that basically gets executed.

B

So as soon as you have that wired up, then this folder under add-ons is what you need to create to be compliant with the other errands and then there's a another part here which has the assets defined.

B

So this is a generated file that gets generated from this add ons dot that go so this is where you define like. Let's see the where the divisor is.

B

Things are so here it is. It's basically just lists all the the the files that are required and for the add-on to be deployed, and then these get basically added into binary data inside the the mini cube binary itself. So when you run mini cube add-ons enable then it will copy these files from the binary itself and run cube, CTL create on them. So the concept is pretty simple: you just have to have this bunch of files that you want to create and in the in the in the folder.

B

If you need some code for that to be built, then you can. You can add that into into the code here as well, so yeah, it's it. It's pretty simple and self-contained.

A

For most people have making an add-on, it's actually pretty trivial, but since this one is basically like replacing the engine, while it's in flight things get a little bit more complicated. When you start messing with.

C

A

Time- and so we had this question ourselves a couple of weeks ago and yeah- look at the G visor, because because we we were like, can we do this in an add-on and the.

B

Answer is yes. Yes, this one is actually a super privileged. You don't run this on that on a production cluster. Yes, sorry.

A

E

So some of our applications, you know, are pushing tens of thousands of lines of code and doing a rebuild is just not tenable for a lot of our developers. You can't wait a minute. Is there any interest in adding like async rebuild command? So if we sink a directory rather than actually rebuilding the command? Just issuing you know a go. Dep go build within the container after a file is synced. Would that be something.

B

That would be with scaffold scaffold, yes raise the PR. We are interested in everything performance related there, so we are looking into two to go deeper and have a tailored experience for language communities to make make development faster so that that is what scaffold is about so yeah that sounds. It sounds like an interesting way to speed that up. So if.

A

You wanted to do this outside of scaffold, so mini cube. Does support mounts but I've also heard people running sync thing between pods and their workstations, which sounds crazy, but you know whatever works.

B

Yeah, it would be definitely nice to create native tooling around a built in tooling. So you can you can do these things somewhat manual script. It up could.

F

You talk to what you see is the big rocks for enabling the offline install I.

B

Think that means a bit more analysis on Aaron to actually to map those rocks out. One of the trivial things I think that there's a pause container that always gets still downloaded, even though that needs to be cached, it should be catch, but for some reason it always wants to download it. There's not that many issues so as soon as you spun up thing, I feel like that. For some reason, the API server has some something as well. I, don't know, I was experimenting. We did, but I have I. Don't have full answer to that.

B

A

B

Have made that money yeah.

A

B

A

The guys at Red Hat have written up some document on like how do you hack, mini cube to be offline where I would like to go with this. Is that the first time you run mini cube that it you know, download it caches everything, and that's so subsequent runs you, don't you just it works by default. That's where I'd like to aim.

F

So I work in high compliance environments where they don't have internet access from their development machines. So even if we could just point to like a local registry or something and just pass that in via environment variable that might be a-okay.

F

C

I just wanted to share more of a comment so when I started my journey with kubernetes, of course, my first go-to thing was many cubes but I feel like the rocket you.

C

Should the users face being pretty small the moment we put even something bigger than nginx container, it starts crashing and there are so so then the person who's trying to container Ares or write specs is confused, am I getting errors with my local community setup, or is it basically so basically that naturally moved me away from many cube to use, cube, 80mm, oars and all I was trying to do, was set up a Jenkins container and run bunch of hello world echo statements, but even then it doesn't scale.

C

So if we can do something around that, that will be really useful and of course, I want to support. Well the gentleman's comment before where every time I don't. If I delete, RM minus RF home, that I did not many cube, we can just have that is or lying somewhere else, which is what we want to do and not download it again. Yeah.

E

It was pretty painful.

C

I did that here twice so I'm, just sharing that, especially if you are in between places and not on your home or your words, you.

B

Don't want to do that again and I think that the Jenkins issue that you're referring to is is related to easier debug ability like in in case you. You want to run a bigger application that fits into the default size. Make you VM. You should have sensible errors that are actionable and say that hey your pods are getting evicted because you're crossing this pressure, you shouldn't have to go into journal CTL on the cubelet logs and then try to parse that out yourself what's happening by.

C

The way I was running it on my macbook, so I'm in a complete like that was even worse to debug, so I am basically gave up. One last comment: I know you mentioned you need help with all the stuff. You said: how do we get started.

B

Great question so reach out: we will. We are available on slack on the the public meetings as well. Come let's have a discussion. Let's brainstorm, look at the issues as well. There's there's like a bunch of issues outstanding. Probably someone else already run into the issue that that you are into we have a pretty large user base.

B

So if you find a relevant discussion, you can join in say that you want to help out, as as, if you, if you create the PR like, would it be okay and then or just do it and then and then and we'll start a discussion there so just go in there download the code base make out / mini cube single command. It's just like makes it so it's relatively easy to get started and start tinkering around with it. So yeah I think that this is this. Does this answer your question.

A

Your machine just kernel, panics.

F

A

Other questions it.

B

Did yeah it says.

C

B

It's done, Wow.

B

G

Free to burn my question after because it's kind of specific but I'm trying to get mini, cube, running on a UNIX machine and since I'm, already using a bunch of stuff, with key VM I'm trying to stay out of the actual box or via etc. And so my understanding was that I have to use the Commission driver. That's going to use Theory search for the network. Part I kind of went into a goose chase on that and I was kind of wondering. Is there? Is that kind of issue officially supported?

G

And where would you what were the key words? I? Should google for in the repo and everything to get me start on the right track, knowing that I'm comfortable with key VM and stuff, or we have a bunch of key VM machines running so I'm I'm, trying to just get into the right direction? So.

B

Let's rephrase it again because I was a little bit distracted by my so so you have a VM and then you want to.

G

I have key M I'm comfortable with qvm I'm running a few custom boxes with key VM I'm, also kind of comfortable with cube black I got a little control pain running in one of these key VM things, but I would like to get all the goodness of mini cube and I'm, trying to figure out how to make that work together.

A

B

We have the key VM to driver I, wonder if you tried it out.

A

G

A

So yeah the KPM driver does use libvirt right now and it should just work if you install the package. I have heard complaints that Arch Linux was just with switched to shipping with a libvirt that doesn't work like for nine or something but I actually use like KVM is my main usage for mini cube. So it's where not only do we test it every hour, but I use it all the time, but let's I would say open an issue for what's not working and we'll we can solve it together.

A

All right, well, I, think that's it! Thank you all for coming we'll.

C

A

Around here, if you have extended questions, thank you. Everyone.