Kubernetes SIG Testing, 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Testing your K8s apps with KIND - Benjamin Elder, Google & James Munnelly, Jetstack.io

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Testing your K8s apps with KIND - Benjamin Elder, Google & James Munnelly, Jetstack.io

Part of the promise of Docker is being able to run tests in the same environment as production. For applications running on Kubernetes, though, development and testing is still a challenge. Developers of these applications have to choose between three poor options: an external, stateful cluster, spinning up a cluster for each test, or running tests outside of Kubernetes. All of these have significant drawbacks that hinder the acceptance of Kubernetes in new environments. kind presents a fourth option: a small, compliant Kubernetes that comes up in one minute, not twenty. Using kind, CI signal can be more reliable, integration tests faster, and local development streamlined.

https://sched.co/MPYy

A

Hello good morning, good morning, good to see y'all here nearly full room. I hope you enjoy the the next talk. It is by James Minelli, the author of cert manager. Do you know cert manager, many of you use it until kind. Do you use kind been elder? Is the author of kind and the ban and James are gonna? Talk about testing was kind.

A

Some of it will be a little bit about. What do you do for certain manager regimes, yep, yeah, yeah,.

B

A

Just just just wondering like how many of you use kubernetes in production and think that you have your testing store stories sorted out: okay, yeah, but, like generally, how many of you use it in production and such alright okay cool, and how many of you first time at cube. Con Berlin, okay, cool! Well, welcome all the new timers and the old timers and over to James. And then you.

C

Go yeah thanks very much for coming today. This is a very, very full room. I can still see more and more people coming in we're gonna be talking about testing kubernetes applications using kind today and a bit more generally too. So, first of all, what is a kubernetes application? I realized that it's kind of a very loose concept yeah.

C

We could be talking about a thing, so it's pretty much anything that runs in kubernetes, but we're going to be putting a big focus on extensions and controllers today in this talk so who here has kind of looked at running operators extending kubernetes anything like that?

C

Yeah nice, like cute builder and operator, SDKs yeah. So this is a good tool for you, so yeah different types of tests. First of all, is something I'd like to define, because I think the difference between unit tests, integration tests, end-to-end tests, it's kind of a very undefined thing: I've never really come up with a good, strict definition. So I'll start by saying unit tests. They typically would make use of something called the fake client in client go. They are it's effectively an in-memory implementation of the kubernetes api server and client. It works great.

C

It does have a lot of caveats. It's very simple! You don't get the full admission control chain on the API server side. You don't have additional controllers running you're missing quite a lot, but it's great for quick, simple unit testing yeah. One big thing that I found is that race conditions, timing issues with the actual controller reconciling and having to run multiple times. You don't pick up on that. So yeah you it's great. It's very good for testing kind of state machine style actions.

C

You know, given this input state, what is the output, but it doesn't really test the whole system as a whole.

C

Something like this. This is an example from cert manager's code base. We obviously build more packages around it, but in short, you can see here. The difference is we've got this challenge presented. True is the starting state and then we actually check. We have expected actions, so here I don't know if I've might have a pointer, nope I, don't have a pointer, but you can see here. We've got the new update action, so we're expecting an update to be we're expecting an update to happen.

C

That's like against the API server and we're expecting that state to be set to false. So you can see here, given that input state, we expect one running the reconcile function once that will end out with a different output state.

C

So then integration test, so who's used cube builder here or looked at it whew. Okay, so within cube builder, they actually provide a load of extra testing functionality. So it will give you a bit of a framework that will start up at CD and an API server and optionally I think even a controller manager on your local machine compared to unit testing. That does give you admission control. We now have an actual API server running.

C

We can pick up on timing issues because this is kind of a bare-bones, so I'm not going to say Kiva Nettie's cluster, but bare bones: keep Nettie's control plane right there. So that's yeah. They work brilliant and it gets you to the next next stage, but there are more things out there to test. Then. Well, there's more to it than that, sometimes with all these kind of integration unit tests, you find yourself tying itself to the implementation of your particular controller, rather than the actual behavior that you're looking for.

C

So that's where end-to-end tests sort of come in and I'm sure, we've all heard the term end-to-end tests. What we mean here is actually bringing up a cuba, Nettie's cluster, deploying your application and actually encoding what we expect and want users to be doing into our tests so saying if I apply. This I expect that, after some amount of time, two pods won't exist and the kubernetes code base is full of end-to-end tests. You've got conformance tests, all sorts, and these are very blackbox.

C

You know you, you go and apply something, and then you wait to see what happens and if it doesn't do that, then that's a failed test. It gives us a lot of flexibility, you're, really looking at this from the users point of view and they're really valuable to have yeah it's black box and everything else.

C

The one downside here is, we need a whole cube, Nettie's cluster and there are lots of different ways to start cube: Nettie's run kubernetes, you could spin up a cluster on economy on a cloud provider or something like that and run tests against there, but over time, if you're doing it's on every pull request or every change that gets more and more expensive and I think kubernetes project itself does so by running standing up GC clusters for a while, which again costs money all of its slow I. Think then you probably know it's probably Bana.

C

Was it an hour hour and 15 to actually go through that whole process? Yeah.

D

Most tests on Cabernets pull requests take about an hour plus to run yep so.

C

You can see that's kind of if you want the open your fast iterative feedback loop that doesn't quite cut it as a developer. I have never run that test suite from my own local machine because, like I, don't really want to dig into how to do it and I also don't really want to run some cloud, some cloud VMS for that period of time and also yeah who pays for that so yeah. Why and to us, though, and to end tests, though some of these edge cases?

C

Some of these problems only come up in a real environment, so you don't actually get to see issues with say running out of memory, because your applications done so like there's all sorts in there that you might not pick up on and yeah the way that other components in kubernetes interact with your own application as well, can make a massive difference, something that passes unit and integration tests in isolation.

C

All of a sudden, you realize that one other cool kubernetes component is fighting against you, and so you end up with this flapping back and forth, and that can go on forever and that'll. Either generate tons of logs use nodes of memory or it'll, just it's just doing the wrong thing: we're just causing excessive load on our clusters. So without end-to-end tests at least I've found everything looks like theoretically, it's perfect, but users are opening issues saying like you kind of just killed my API server. Nothing will continue running I can't start pods anymore.

C

So we have an example of where this came in really helpful. In certain manager. Recently we've gone and refactored some API types.

C

We needed to make sure that as part of that refactoring, we didn't break the old API format, so I started out by implementing the changes so actually implementing the new format of API I didn't touch a single thing in our end-to-end test suite and pushed all that code up or ran it locally, run the test suite locally and made sure that those tests still passed and because they are blackbox, they don't understand the actual implementation what's going on if they pass.

C

From my point of view, that's like I just ran users like users, just sat there and ran this same flow and the fact that that passed gave me confidence to know that I could actually move forward with doing this. Of course, I then went and added new tests at the end, because you can't just add new functionality, but yeah you gradually grow and grow your test suite and as you're doing that it bill your confidence, and it just means you can actually ship faster.

C

We've done this countless times before before the test this week and where you think everything's perfect, it turns out it's not quite so much yeah because it says they give you confidence to ship without it. Who knows what will happen so that leads us onto kind. So we've been talking a lot about how difficult it is to get these clusters. Then.

D

I think you've got today, so hopefully James has convinced you that into and tests are definitely something you should be thinking about, and if you're thinking about into and tests, you should think about kind kind, is kubernetes and docker, so we run docker containers to simulate nodes. It does multi node multi cluster works offline. If you pull the node image first and if you're, putting a single node with the image pre-poll should take about 30 seconds to boot up a node.

D

So this takes away a lot of those problems and it hits these key goals a one if you're gonna be doing testing, ideally as much as possible, you want it to be fast, so you get that iteration cycle.

D

You want to be lightweight so that you don't mind running it all the time you want to be cross-platform, so developers can run it on their machines, doesn't matter where they're at we're gonna be hermetic, which is a term I mostly heard at Google, but the idea being that we don't want external dependencies because those will decrease the reliability of your tests. You just want to be testing your application, not the internet, so kinds of testing tool, typical workflow, Buddha cluster, build your application and inside load application images. We're gonna come back to that.

D

You don't have to do.

C

D

Way, we're gonna recommend that then you run your tests do again. Creating a cluster is just kinda create cluster. You need the kind binary and you need docker. That's it.

D

Sometimes you might want custom configuration if you're doing this with kind. You can configure the entire QP item interface, it's Q beta, underneath we have customized caches, which we're going to show in a second, and so you can configure anything. You need to make kubernetes look like how you want it to look, and that can just includes things like changing the service cider or enabling helpful features, anything that you would expect to be able to do with a fairly real cluster. So this example actually from some assert managers testing.

D

So what you're seeing here is kind zone configuration so we've got this just configure object, I, believe it's cluster now, and so we've got one node and on that node we're saying we need to patch the Kuban config the set of particular service net, because kindness, herb sorry, cert manager is doing some fun things with networking, because they're doing like DNS challenges and whatnot, so they're, very particular about which subnets things are on. So you can just configure like that very simple or if you want a bunch of nodes, you can have multiple control.

D

Planes will set up a load, balancer container and do H a and you can have workers, and you can simulate doing like rolling things across your cluster between nodes and if you want to do things like okay kind, ships, its own lightweight, CNI, but I wanted to integrate with weave, then you can disable the default. C9 deploy your own.

D

So now that you got your cluster up and ready to go, what you're gonna want to do is load your image into it, and the idea here is again avoiding flakiness, avoiding being not hermetic, no dependency.

D

On the internet, instead of pushing your image to a registry in your testing flow, what you do is you can build it and then you can tell kinda load it into all the nodes, and so basically it's just going to take a docker archive of your images and then stream those to all the nodes and load it in, so that when you tell kubernetes to run your image in a deployment it's already present, it doesn't need a full. You don't need to use the Internet. You can do this offline.

D

So now you're gonna get the cute. Config kind generates a unique cube config for each cluster, so you can ask kind where the cube config is point cube, config, environment variable at that run, your tests now dreams, gonna, do a live demo sure this will go smoothly. Yeah.

C

Cool- let's see, hopefully go a little bit bigger, hopefully at the battery web, see so in this demo, I'm actually going to deploy the sample controller from the cupola to use code base. So this is just a really basic example: I've put together a very, very basic run test script, so here we can see. We do first of all a little bit of cleanup, because I had to do a bit of testing here and then we're using this client create cluster command.

C

That been mentioned earlier well go into and have a look at that config file in a second, but once that's run just like Ben showed we go and get the client a cube, config path notice here that we're passing a name variable. This is because we can run multiple clusters at once, so, for example, the federation v2 project, they utilize kind to actually test federating multiple clusters. Together they pass different names for each one and yeah and that all runs on one machine here we then go and build.

C

Our application so obviously depends on what you're doing here, but in our example we're doing sample controller we've built a docker image for it and then just like Ben said we load the docker image into the node. Now, if there were multiple nodes, I think in this example rowing into a single one that would actually get loaded into every single one of those nodes. So it's available for use, we deployed the sample controller and then we'll start using it so enough.

C

Looking at scripts, let's bite the bullet and see how this goes so do a bit of cleanup, and here we go kind is going along and it's creating a cluster, so we're booting version. One point fourteen point two: this is just using one of the prepackaged ones that Ben has pushed up forever on.

C

It prepares the nodes. First of all, so that's creating the containers. It then goes and initialize queue initializes cube, Adam.

C

It applies all those patches that we've specified onto a base config and then it will start actually initializing so that under the hood is running, cube Adam in it within a container we've already gone and configured systemd, docker and the like, and there we go, we install see and the class is up and running so we're on to the next bit our script, going we've built sample controller there, it's getting a little bit too quick and we're building a docker image containing it, and you can see here.

C

We've gone unloaded that image into the client container and we've gone ahead and actually deployed the sample controller. So in your application here, obviously the manifest you use the commands you run will be slightly different. We're just trying to illustrate what a typical workflow would look like that at the end, we've actually gone and managed to deploy one of our few resources. This is an example. It's not particularly useful and you can see.

C

We've got our command to run here, and this is as been said, the hopefully you can see at the back this here is the cube, cube, config file. That kind is created. This is typical works with everything everything I worked with keeps ETL, and we can now go ahead and check our cluster. There we go so you can see just like any other kubernetes cluster. That you're used to we've got all the different components. We've got our CNI here.

C

We actually run a custom lightweight CNI, that's perfect, just for testing on a single host, because we know where we're running. We can understand the network topology better and we've actually got our example application here, our sample controller here. So, as you kind of continue to build now that we've got a cluster up and running, you can continuously run that kind, load, docker image command and load more and more images into your nodes.

C

When you roll out changes and then just restart that pod and off you go so this gives us not only a testing environment but also a development environment here I'm running on my laptop, but this could have run under CI and it will be identical. The fact that it is identical is so valuable like if you have varying, if you have varying different environments between different developers and everything else, it's frustrating we go. You know you see, your tests are failing, but you can't possibly actually debug it because you don't know where to start.

C

You don't know how to actually get this up and running. So we haven't I will clarify we've, not written a full test suite here. This isn't telling you how to right tests in bash. Please don't write your tests in bash. Anyone really doing please yeah.

E

C

Looks like didn't go wrong and yeah, so we are running here kind. In on my on my laptop. We do support running kind, obviously in various different CI platforms, just put together a kind CI, slash examples repository, so that's coming together. Gradually we currently have some examples for running kind, on circle, CI and Travis, gradually, adding more there, but technically anything. That's Linux, your pretty good to go or Linux OS, X or Windows I should say probably.

D

The most one- that's not in this repo yet is kubernetes its own CI, which is of course running on kubernetes. So we run kubernetes on kubernetes because who wouldn't want to do that? Yeah? You.

C

Can go many layers, deep yeah and one extra thing to note here: we've kind of used the CLI side of kind. We do also try and like we want kind of itself to be consumable as a library within go line. So here's an example where we actually go and create a cluster all in code. Now anyone who's familiar with golang testing.

C

You can just go and chuck this into your test functions and stand up a cluster, that's used for a single test, and that I mean you can see the clusters boot within about 30 seconds, but we can actually just bring up a cluster for each individual test, which means that you know that if it passes once it should pass again and again because the cluster bring up is, as Ben said, hermetic. So we know we don't depend on any external sources, yeah, no more bash.

D

So I'm gonna talk about. What's next we're hoping to get better network management, we found that you know we want it to work perfectly everywhere, but one thing that is actually different for everyone is your local DNS on your host and that gets picked up by kind right. Now, we're figure out what to do. We'd, love to hear ideas, ipv6 support and testing so kind was actually originally written for kubernetes testing of kubernetes the project and we don't have any ipv6 CI.

D

So we're working on getting ipv6 worked out in kind, so we can get single stack and then next will be dual stack as kubernetes starts to grow that we want to make note ports easier to access on Mac and Windows. This is a bit of a caveat for application development.

D

Now, if you're on Linux, all of the nodes have IPS, as you would expect that you can reach because they're just docker containers on the docker bridge, but on Mac and Windows they're inside, like the docker for Mac theam, which doesn't expose one so you'll have to port forward we're still figuring out the best option there. You love to hear feedback there.

D

We want to make it easier for you to build like an exact kubernetes version, but chiefly kubernetes on CI is building tar balls all the time that you can just download it like whatever version you want, so we want to just take that pack it into a kind image. Then you can run it we're always trying to make things faster. We have few more ideas there and we're trying to move from alpha to beta very soon. Hopefully.

D

But if you use a release version, we probably won't break you and I'd really love to hear from interview your use cases and like what features you'd like to see because it's a small project right now, but there seems to be a lot of attention, we're trying to figure out what a prioritize so appreciate that here's, where you can contact us. This is under the sink testing and the kubernetes sub project.

D

We have a bi-weekly project meeting on zoom' like the most kubernetes projects, we're on the sig testing mailing list, and we have our own slack channel on the kubernetes slack you gonna invite that's like buckets. Do the repo at 6:00, our kids, do / kind, which is also you can just go, get that if you have go modules on and the docs are it kind of sig stuck he'd say?

D

Oh thanks for coming and I want to do a special shout out to Liz, frost and hep tio, who actually put together the CFP but couldn't come and James for jumping in and helping square this away, rework the talk of it and bitching, and thanks everyone for coming.

A

Thank you very much thanks for making kind such an awesome project and well, we do have plenty of time for questions actually and yeah. Please raise your hands and I'll run over to you as a mic. Okay, so we got two hands here, one there. Let's start here.

F

Yeah um ipv6 support hasn't been merged already in the kind or it still exists as a extra PR, it's in a PR, it's almost there I'm gonna be reviewing that some more. When I get back I mean. Could you be a bit more specific for the time when it's gonna get merged? It's kind of important and we're.

D

F

The same state about month ago, so right.

D

It was for about two months. We had a bunch of other chaos going on and we got back to that. We've had a lot of progress on it recently, but we're not quite there were very close. It turns out one of the components we're using is not working in ipv6 properly we're gonna swap that out and then I think we're good to go, but she's.

B

D

Of a filling one, we're hope when it's ready we're hoping for another release about a month from the end of cube con, we try to do monthly releases. We did release just before cube con, so hopefully that one okay.

F

And sorry, second, one very shortly, let's say if my PRS actually changes the API server. What would be my workflow to use when, basically to use the custom API server image when it when your lustre comes up, kubernetes, PR yeah, you.

D

Just need to do slash test, pull kubernetes kind et with hyphens between them and we've got an optional pre cement. Now we're hoping to probably make that a default one soon.

G

Do you have a plan to support, creating image retreats and share the volumes, such as universe or ice cozy around with creatives across sir.

D

I'm sorry I'm, not sure I understood the question so.

G

I create Cresta I want to create a registry as well, so I can just pushing images from humanities using dictor or 1080 wheel and I also want to create shared volumes such as I knew it NFS or Isis target I wrote with the cross so.

D

We we do have support for adding extra volumes to the notes to pass through things. We do not have any special support for registries right now, but it is something we're tracking.

H

Just go in order.

B

1 is it possible to use card with another container, real-time like competitor de or cry, or something like this using.

C

Kind of the other controllers not.

D

Yet we're looking at that we, so that's a big thing that we want to do. Probably after we finish. The ipv6 stuff is a refactor out, so that we're not talking to Dockers in many places, and we have like a little kind of CRA Ashim for unit testing and pod mana will be the next one, we're targeting CRI functionality as much as possible, like we don't use excessively docker specific things where we can avoid it, but it will take some time, we're big fans that we're running container D on the nodes.

D

Now, because we've been able to do a number of optimizations around how the food up happens because of that and I'd like to see more of that all.

A

Right one more question: here: ok.

I

Since you're suggesting that we should use it as a library, you not use bash and you're still alpha. Do you have any plans on how you're doing versioning with kubernetes? For example, you have a project using 1/12, typically, if you're in education or something you are a little bit behind, so any plans on how to do that in in future.

D

Possibly using replace um that's a really tough one and that's part of why we focus on the command line first, because yeah that one's tricky. Why not one solution is to just have your test suite import this, and hopefully your test suite can be decoupled enough to just talk to your application, and then you can version that separately, yeah I.

C

Think I think that comes about kind of from the blackbox nature, for example, or our test suite never really touches any core application code. It's it is basically its own application. We compile that to its own binary.

J

I would like to ask: is it possible to deploy images using hell.

C

You mean the the docker images or deploying.

I

The applications on it- yes,.

C

So I mean you can yeah, you can use helm against the cluster helm and teller against the cluster. That you're deploying to, in fact manager does exactly that. So we can verify that our helm charts also work as part of our end-to-end testing flow, the docker image side of things, I, don't think, that's something that helm does generally anyway, publishing images using helm, I think it's more as the deployment, although lots of things change in this space quite rapidly, so I'll disclaimer that okay.

A

Thank you, okay time for questions still any more questions on this sides, one down at the front, one down in front all right.

K

If we already use prowl at the CIE, would you recommend a starting path to use kind.

D

You should send one of us a ping we do have that set up, but it is a little bit hairy. Might give you a hand with that to.

C

Clarify we use prowl as well, and yeah I've been running kinda nerf, quite a while. Now you just need to make sure you make your pods privileged enough.

D

I

D

Else kinds, own pre, submits, run probes. So we can point to that.

A

Another question here: okay,.

L

Hi any plans or work already being dumped to use kind as a library from other languages.

L

D

Think not I'm hoping that I mean so the command line is meant to be used from these things. But that's an interesting topic. There's a bit of logic in the kind library to do some of the setup. We have to do things like generate the config or like we used to do more things during boot, up to control that we've moved a lot more of it to be basically just running the image. But there are a few things that we have to set up and configure and whatnot.

D

We basically probably have to like rewrite the kind aspect in another language and.

C

I think, maybe over time it could be more possible if we manage to reduce the amount of stuff. That kind itself has to do until we can just get it to the point of a docker run, I mean I. Guess you can rap, you know right, docker run in Python or something perhaps I was.

A

Thinking, maybe that there's the kind cluster API provider, which turns kind into something that you can access with cluster API and after that he could use any other any any client. Any kubernetes client attention to to to to do things with it. I can't speak of the maturity of the project.

D

Or anything like that, we have actually discussed that it gets a little goofy, because right now, cluster API uses kind to bootstrap by default. So would you like bootstrap in kind with kind but sure theoretical, purely.

A

Theoretical idea so there's another question here:.

M

Yes, I was wondering how do you approach or if you have any plan to approach the problem of external dependencies, for instance, let's say the case: I'm I declared a PVC using a storage class which is a natively, a storage class because I've got an e KS, managed cluster or I'm, using an ALP, ingress controller or even I'm, using an external database. Let's say dynamo DB so.

A

So you would want to sort of mock it out.

D

Its storage is a particularly tricky one with kind I think you can install a fake storage class that pretends to be the same, but like some level when you start to get pretty far in those things, I would recommend, like you, have a QA pass at some point further in the dev cycle, where you actually run in a more broad like environment and make sure that everything works out is expect there. Because storage is one of those points where like if you're not specifying many things, it's gonna work great and be exactly same everywhere.

D

If you start tuning things and whatnot, is it's gonna be a little bit different, that's hard to avoid.

A

More questions, please: we've got five more minutes: okay, one over there I'll try to come over all right. Okay, somebody.

J

Hello, I have a question more about testing operators and how you do this internally, with kind and particularly with eventual consistency. So the problem we are having, as our operators are just eventually consistent, so you run into like flapping tests because you have a time limit set. How do we handle this internally or do you have any wise advice to do with this yeah.

C

That's a great question: that's we had a good discussion about this the other day in some colleagues. So, like you say, unit tests, don't really pick that up, because you run a reconcile function once and actually things are eventually consistent. We need to keep going.

C

This is where I'd say, kind really shines, because, obviously you are running your controller. You are doing that whole thing.

C

You do need to be careful to maybe count the number of times it takes to get to your finally consistent state or how long and that can generally be handled quite well with test timeouts and the like, I think some of the other strategies on a more like a unit testing level could be considering running kind of the reconcile function in a for loop up to 10 times 15 times 20 times, whatever is appropriate for your code, yeah that that's what I'd say for the eventually consistent things, but with kind itself.

C

You do kind of want to do this in a larger cluster because it may be that you're not actually getting to your consistent state, because something else is fighting you basically yeah.

N

Does kind give you an ability to restart your entire kubernetes environment between tests or between tests. Suites, do you mean reusing the same cluster all like? Let's suppose that I want to clear my state between the test, suites and maybe clear some kind of database state? Would it be possible to restart it so grammatically.

D

We actually recommend just throwing away the cluster and making another one it'll. Take you about 30 seconds to boot occult, but.

N

Like would it be done in an external script or is it like? Can it be.

D

Coded into my test, like in like in your test, harness like okay, so this test, these set of tests are gonna, run with this cluster, and then this set of tests are gonna use. A new one, just spend one up: tear down tear down is near instantaneous, it'll. Take you a second or two, and up is about 30 seconds. So if you really want that clean state, there's no better way than just actually having a clean slate Thanks.

A

Someone he had a question.

H

D

Alright, if that's all the question, oh there's one, there yeah.

A

D

So could you raise your.

N

Hand again, please.

E

So so so time to ask really.

D

Quickly, I want.

E

A heist on stage.

D

Here anyone who has sent us a PR would you come up. I know: they're, a bunch of people are here, I see you Antonio.

E

Okay, the question would be you mentioned that you also run in Windows, so using kind as a deaf environment for like a prod like environment or to work on Cuba natives on Windows. What is it possible, or do you what? What are the major problems surrounding this when using kind it.

D

Should run fine because it's just using docker containers. That said we don't support Windows nodes yet, but we do support Windows as a host. It's a little bit not quite as well supported, because we don't have CI for it yet and I don't develop from it, but I do periodically check and make sure, and we avoid anything. That's operating system specific on the host side. Thanks thank.

A

You Thanks thanks very much. Everyone.