VMware TGI Kubernetes, 29 Jan 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: TGI Kubernetes 144: Exploring The State of K8s on Windows

Description

Come hang out with Jay Vyas (jayunit100) and hack around w/ windows and various CNI providers in EKS, VSphere, and beyond. We'll also look at what running containerd in windows looks like, and go through the basics of "omg, im so used to linux, how do i do this stuff in windows?' for those of you in the K8s community that are just getting started with Kubernetes on Windows !

Show notes available at https://github.com/vmware-tanzu/tgik/blob/master/episodes/144/README.md

- 00:00:00 - Welcome to TGIK and introductions
- 00:00:10 - Week in Review
- 00:15:18 - Looking at how NetworkPolicy Truth tables work on unsupported clusters
- 00:32:29 - Viewing working NetworkPolicies in Windows with Calico 3.16
- 00:45:01 - Lookint at CAPI on Windows
- 00:55:40 - How CNIs are installed with post-kubeadm/preBootstrap commands on Windows
- 01:02:15 - How runtimeClasses, taints, nodeSelectors work together to schedule Windows pods
- 01:13:53 - How Cluster API works on NSX for windows with VSphere and NSX as the network plane
- 01:15:34 - How to use hub.docker.com to lookup windows images matching your OS
- 01:19:07 - Looking at CSI Proxy, briefly
- 01:26:45 - The most interesting problem in containerd and windows networking : The CNI ADD Codepath!

A

Okay, do you all hear me.

A

A

A

I'm assuming you all can probably hear me now, assuming you all can probably hear me now.

A

Cool, oh great, okay. I turned my audio off all right, okay, so so um all right, let's get started yeah, so um I got a it's my first time streaming doing this stuff. So uh let me grab this markdown that I have and bring it over here and so hey everybody. What's up, okay, gab, I see gab is here I our csi hero. I see ricardo claudio um a lot of other people, martin some of y'all, I don't know tim um cool. What's up yash antonin from the andrea, we got yash from synopsis.

A

This is great okay. So if anybody has, if I you can't see anything just whatever shout out to me, let me know um so: here's what we're going to do. um It's tgik!

A

So it's my first time to host I'm jay over at vmware um and um we're going to start out with news of the week and if anybody has any other news that you want to talk about, feel free to just like post it. um So we'll start out with those of the week and then news of the week and then we're going to go into um some windows on kubernetes stuff.

A

Some really like deep internals, looking at container d on windows, which is kind of new, looking at andrea and calico on windows, um which are also kind of when it comes to container d, pretty new as far as cni providers and we're going to see something real cool that mark rosetti showed us recently where you can actually run privileged containers in windows using a couple of hacks. Now it's the first time I think anybody's been able.

B

To really kind.

A

Of do that in the upstream in an upstream capacity, so it's really exciting um and that's pretty like cutting edge stuff, so um yeah to start out um and thanks to everyone who helped out with this csi stuff from gab perry has worked real hard on getting a lot of this stuff working the android stuff, especially and um the image builder stuff, which I probably won't have time to demo um and, of course ravi from red hat.

A

He gave us so he gave us a nice sequence diagram that I can use for some of this stuff to show you how container d works. So, let's go okay, so uh announcements of the weeks we got cka. I don't know how many of you all are taking your cka.

A

um Let's see, oh yeah go: go full full screen and increase font size. Yes, sir good idea. Okay, so let me know um if that's better there we go there. We go okay, cool! So, let's start off over here we've got cka ckad exam tips from brad brad mccoy- I don't know brad but sounds like he's- worked pretty hard to put some stuff together for us. So, let's see what we've got, um I never took my cka.

A

um I never took it. I never got it. I don't know um so I I never took it, but I heard it's getting harder and harder and I knew some people that were really really really good at kubernetes. That said, it was hard like people that were writing cloud provider stuff and writing cluster api extensions and stuff. So it's no joke. um Okay, so he's got. This is cool, so he's got. um Let me make this a little bigger.

A

If I can there we go um and I am getting I'm starting getting the rhythm here of looking down at this chat. So you know if there's some latency before I see you all, leaving some feedback feel free to, like repeat the same thing: okay yusuf, I got you I'm gonna, I'm gonna hook you up with the big screen now: okay, there we go cool, so okay, linux stuff by the way, we're gonna, see all the equivalents or some of the equivalents of these for windows.

A

Today, there's a lot of really really cool powershell commands that you can do that. A lot of folks. Don't really know I didn't know it took me like a month and a half to learn them um to do this kind of hacking that we're used to doing in the linux terminal. um So, okay, linux commands for the cka control r. um You know who showed me control r, like eight years ago, was jeremy eder over at red hat. So shout out to him for showing me control r. I never used to use it.

A

I never even knew why people cared about it. Okay, make use of the watch command. um Let's see who, um let's see what else we got here: coop ctl api resources? Actually I didn't even I've never even used this command before so that's kind of cool okay, so it gives you, let's see he's going to run it for us in the um in the animated giphy terminal right here. um Let's see here, better, okay, cool yeah, so get names short names.

A

Okay, this is yeah. I always screw this up. I always guess like ds, I get lucky. um Np doesn't work for network policies. I've tried that um okay, nice coupe ctl run. I always get confused with this because they changed it. It used to work and then it stopped working and there's some new way to do it. So this is all stuff that anybody I think going through. This would be great for anyone. I gotta I gotta jump, though, because we got five more of these. Thank you david um and uh kubectal plug-in.

A

Okay. Let's do this we're nine minutes in so I'm gonna have to burn through this. Oh, this is youtube video. I don't know what that is. Okay, uh maybe it's the wrong link, but uh I like noun, verbs too open ai scaling kubernetes to 7 500 nodes, okay, so this um so I I was kind of I saw this and I was like okay, so this is for ai workloads. I guess right, and um so everybody- I guess, is doing this. I know alibaba had been doing this and I remember there's like this.

A

There's cube mark that you can use to test this kind of stuff and I think that they found that the bottleneck in this was um the bottleneck in this, as I recall, is the library does anybody know the library inside of etcd? I forgot what it was, um but there's like a library that ncd uses to do raft. It's like some in like embedded database or something, and I thought that they found that that was like the biggest bottleneck they had and they. um So let's see what these folks did.

A

um Okay, iptables rule iptables rules, um maybe yeah. Okay. We know that that doesn't scale. We know we know. People like andrew um over at sig network are working hard on making ipvs better for folks. um I don't, let's see what else they did, I'm going through this quickly. Oh api server. So it's the lists right here. It's the list that are killing them. Okay, yeah, okay. We yeah! That's! That's always been that way. I guess so cool anyways, another tuning article great. So this one and then the other one that I really like about.

A

This was the alibaba one. If you guys are interested or if you all, are interested um okay cool, so dave giffin crash loop back off, I'm gonna jump, because no, you know what I just everybody deserves a chance here. Just so you so you all get to see it. Okay, if you use kubernetes while debugging the crash loop back off okay.

B

Yeah, okay, ready yeah.

A

Okay, so this is good. Okay, these all these links will be in the markdown um he's putting what is he putting in here curl now, you need to add the necessary tools to help with debugging proc ps. I don't know what that is something to play with if you've got a package manager that supports it. I have no idea what proc ps does: um okay cool, so cockroach labs, I'm going to skip over that. I know a little bit about the cockroach labs scheduler and I think what it does.

A

Is it's smart enough to switch things around no zones schedule smartly across zones which anybody who's done. Anything with stateful sets and volumes knows that that's a problem. I know that because I looked at this earlier, so I got to read a little bit about it. So I'll just tell you that, um and uh let's see we're already 10 12 minutes in so I'm going to uh oh and of course, james is here from microsoft, with mark and um mark, microsoft have been super helpful with all the windows stuff we're doing.

A

It's really cool so james. We don't need to show the twitter we'll just jump through it, but james just announced that cluster api for azure is now released. Support for that. So that's really cool james. If you want to post any links about that in the um in the chat down here like go ahead and go for it, yeah crash the office npe for for containers. I agree with you. Yusuf, okay, I'm gonna read the chat. I'm gonna take a second okay. You can also use linux commands on windows.

A

Oh yeah claudio okay. Well, but you can't do that on the node, though claudio right, except for now you can because mark rosetti gave us a secret hacked up container d windows, privileged containers, things that we can actually um that we can actually try out today. So um so I I'm gonna do so before we go into that stuff. I'm going to show you all how like the the windows exp. So there's a lot of tutorials about how to I'm going to jump into this demo. So, oh wait. I missed a couple of stuff.

A

We had some other news, so, um okay, so joe, we got a kuba kube academy thing from yesterday. That's cool, so coop academy is really good for, like if you're just getting into kubernetes and learning about it, and um so now we have a new. We have a new, a new video up that um and then let's see, I hate ammo.

A

Okay, I don't hate yaml. I know I know I know some people in this room- probably hate yaml. I actually kind of like it um is this. A dsl software service nightmare is on cloud street. Jumping sorry about the ammo, oh yeah,.

A

Okay, so I really like the history of all the different yaml tools that we've had in the career, including case on it, and I remember by the way when, when, when hepta came out with case on it- and it worked in my ide- I was so happy.

A

um I this is oh okay, an attack surface things get tricky when you start deserializing anything more complex in order. Okay, so now we're getting so this is yeah. There's a lot here, it's too much for me to go over right now, there's a lot of interesting kind of like he's like combining with ruby. It looks like okay um production, kubernetes josh, um who has helped so much get the docs together for this uh series has a book, I'm sure it's an awesome book.

A

um So uh let's just take a look at that and uh what is it? Where is this coming from? Is this a o'reilly manning? Oh, it's an o'reilly book, cool, okay and it's got some vsphere stuff in it. Awesome, um okay, great cool, so we got a new book coming out um harbor to deal with docker hub rate limits. We do that internally here at vmware. It works pretty. Well, I really like it.

A

In fact, the images you're going to see today are running off of perez harbor in harbor uploaded images that he puts up there, which is cool and spring boot apps. So I know we've got some spring people in the audience over there, and so uh this is the last one and then we're gonna go straight to the demos. Cool spring boot springs important.

A

I don't know the. um I don't know how many, how many apps run on it, but like it's like super popular and super important, and I know over at blackduck, it was super important spring startup times were super important used to sometimes take a long time for the containers to come up, um and so we lived that um for for those of you from blackduck that are in here. I know you you deal with some of that, sometimes so um cool, let's go. Let's start hacking on stuff, okay, so windows.

A

So the big thing about, hopefully, what I want to show everybody- is some of the deeper internals of windows today, the stuff that you're not going to see on um in googling for stuff. There's a lot of weird stuff that I had to learn when that for a linux person seems weird because you're not used to this whole windows idiom, but it actually winds up being a pretty powerful idiom to do things in um for one.

B

If you learn how to use do stuff.

A

On windows- you're not stuck with a macbook anymore, um so you can it like, doubles the or it it multiplies by 10 the amount of laptops you could use. I've got another machine over here, a windows machine that I use and I'm going to share the screen for that by going to it over here, so I'm going to make this bigger and then you'll be able to see it. So I so the first thing I'm going to show you um is yeah, okay, so the first thing I'm going to show you.

A

My audio may not come out super clear from the beginning, but it doesn't matter. I'm going to show you a couple of clusters that are running on windows. One of them is an eks cluster. Another is an aks cluster they're, both running calico, okay, so we're going to start out with kind of the standard stuff.

A

These are these are basic clusters and we're going to look at how you install them, how you get them set up um and how you kind of administrate them from a windows perspective right and by coincidence, I'm actually doing this on a windows laptop just just to be windows, so like okay, so um so the first thing I'm going to do is I'm going to set my so I just set my: um can everybody see yeah it's the year of the the linux laptop yeah. So um does?

A

Can everybody see this terminal or do we need it bigger, just yell at me? If you need it bigger, I'm checking this all right. So.

A

One second, all right, so there we go so here so here. What you've got here is you've got a um you've, got a you've got a yeah. There we go. It takes there's some latency for this to catch up. So what you've got here is we're inside of an aks cluster.

A

You can see there's several calico node deployments and running those are all running on linux nodes, though okay, so when you install calico right now on um when you install it right now on on anything really whether it's windows or on any windows machine, whether it's aks eks, wherever um you're, going to be running it as a windows service right. So these are like the linux calico nodes right and um if I want to administer this machine right the way I actually do that from a windows perspective.

A

Let me see if I can make this a little bigger there. We go.

A

This may not be visible to y'all, but um I'm going to see if I can zoom.

A

In ah there we go, I'm gonna check on my stream. Can everybody see that? Can everybody see this what's up ravi? How are you doing thanks for that sequence, diagram by the way, can you see that now yeah wsl has come a long way so on this machine mark?

A

Actually I actually wanted to set it all up it to be like windows idiomatic just so I kind of learned it and I actually I'm going to run the end to end tests from a windows executable in a second um that I built in wsl2, um and I and and so I I found that I was able to use windows for development, which is like for end-to-end development, which I think is cool, um so can can you all see the?

A

um Can you all see the uh yeah okay cool, I'm gonna, assume that you can see it all right. So when you run.

B

A

Yeah, so the first command that I wanted to talk about here is um the hns hnsdiag command. Okay, so this is kind of like I don't know what to compare it to it's kind of like ip tables, but not really because um ipta, it's it's like a it it well, let's just take it from the from the windows perspective. It gives you like a global view of all the hns, which is like the networking system subsystem for, especially for when we're running containers where that comes in into play. Right.

A

There's the h, the the hns and the hcs and those two subsystems are what allow you to like build containers on windows.

A

So I haven't really gone into the architecture of any of this, because there's so much other stuff to go into, but in general, um when you start a windows container, there's a shim and the shim then makes like api calls to like lower level um like windows, lower level windows, services that are built into the windows kernel, and so the two big ones that I find the most useful are running the hns diag and then the uh the other one is the hcs I'll show you that so that well the hns diag before I switch I'm on an aks cluster here you can see it gives me two networks, two top level networks and those are the container networks that like are globally set up for me like on my.

B

A

And then you can see all these and you can see all these endpoints right, so you've got devices for every single pod and then you've got load balancers right, so you can see those load balancers and you can think of those like iptables rules right um and so we'll see that, like you know this, this happens for kind of any cluster. Now um now, let's run now we're just going to run the n10 test.

A

I have a modified end to end test which um the link to it is is in this stream and I'll give it to you in a little bit, but it allows you to run the network policy tests on windows and, let's see what happens like we'll just run it on um aks. It doesn't support network policies on windows containers. Yet I don't think, but I think mark and james. You can correct me probably on the way soon, but we'll run it anyways. Just so we can see running the enten test and what it looks.

A

A

Foreign, I got a lot of leftover name spaces in.

A

A

A

Okay, so we're gonna see here so um again, so something nice just so you know you can absolutely do end-to-end kubernetes development on windows. Here I've got an exe. It's really easy to build these binaries.

A

You can't do some things like you can't build on windows itself, but wsl 2, like mark mentioned, you can kind of do it all on there. So these what I do is I just build these on wsl. This is the end to end test executable for those of you that don't know about the kubernetes end to end tests.

A

Kubernetes have released ships with an end-to-end test library that has 130 or so conformance tests and about you know, like 400 other tests, that you can run to confirm all the functionality, storage, networking, all the other stuff so that whatever provider you're using, we have a tool called sonobuoy. That makes it really easy to automate uh running those like periodically if you want to and so on anyway, so I'm going to fire this off.

A

And the first thing that I wanted to talk about when you run these tests, is you send an option for folks that are running e2e tests on linux? And you haven't done on windows? You send an option node, os distro and when you send that node os distro option, okay, um yeah like 270, that's whatever yeah, sorry claudio this okay, 270 conformance test. um Okay, so uh are those like including storage tests. I could have sworn it was like 200 or something but okay, I don't know- maybe I maybe I omit.

A

Maybe I run them so much that I don't omit like some of the slow ones or something um anyways yeah. No, you can't ricardo no call of duty in the container, so you can't do it. I don't! I don't think you can really do video games like that. Okay, so um yeah yeah, we could do a dry run yeah. We we should just do a dry run and yeah you're, probably yeah you're right, um so uh so claudio's right. We have more than definitely more than 150 conformance tests um and uh wow.

A

I did not know that, but a lot of those are auto-generated right like I ran a bunch of storage tests, and I saw like thousands of tests being generated and a lot of them were not tests that I would want to run. um But I didn't know we were in the thousands okay. So um so these tests are running. You can see these probes being sent out, so you send a node os distro and what that tells the n10 test.

A

Library is that I'm running these tests on windows and when it's doing things on windows, it does things slightly differently. So, for example, it's going to exec into a pod and it's going to run like a powershell command to verify something, as opposed to like a like a linux command to verify it right. Okay, so um so these are running.

A

These are creating a bunch of probes and what the probes are doing is they're, going into um they're going to instead of there's, there's there's nine pods um and each of the pods is probing each other, and so there's 81 possible connections and we're confirming that there's a matrix of connectivity. What that matrix looks like um and there's a couple of modifications I had to make to uh get this to work.

A

um One was make it so that these tests target uh nodes that are labeled windows and then another thing was make it so that they support this node as cistro, but perry. That log message is for you. I leave these in here for him, so that when I go to bed at night, he can see that the the cluster's broke. If.

B

It's broke anyways, so.

A

um So you could see here. This is how you know that you're you're, not supporting network policies. um Again, I didn't bother to set up network policies on this. um uh I think there you can do this on ik. You can do this in azure. If you wanted to, you just run the updated version of calico, but I did another cluster in eks. I wanted to do all the different clouds, so I.

B

A

Any cluster in eks, where I installed my own calico that I'll show you where I did do this so anyways. But what you can see here, as you can see in this matrix, that in windows nodes at least um the connectivity between these things is, um is it is? This is supposed to put a deny all policy and you can see the dots mean that these things can talk to each other and the x's mean that they can't and so in the comparison um in the top.

A

In the comparison, we expect a lot of blocked connectivity, but we see that all communication is allowed, and so the test fails. So that's how these tests work they're, I think they're. You know I think they're good for they're, we're using them to study, network policies and a lot of different providers, and it's real important now for windows, because we want to ship, andrea and calico, and all these other things. We want people to start using network policies with the with those cni providers as well on different clouds.

A

um So uh I actually have a question for our microsoft. Friends who are here, which is, uh are you all gonna up? When are we gonna update the latest calico in aks, um and then uh that that would be an interesting thing, because once we do that, we can. We can actually do this. We can actually click on the buttons and get a cluster that supports these policies but anyways.

A

So that's the first test I'm going to do now, I'm going to switch over to a calico cluster, that's running on windows and then we're going to dive into that with rdp, which is so. We can poke around and see how you install the latest version of calico. That does support network policies on any windows node, and I presumably you could do this on azure or or or or or an amazon cluster. How aks and eks work, I'm not 100 sure, but I'd be interested in knowing more yeah.

A

A

Okay, so I'll run the same tests here right and.

B

So now I'm running.

A

The same tests over here and now, I'm I'm I'm in this rdp panel to show you how it all how some of this stuff gets rolled up into windows. So I've already showed you there's an hns, diag and I'll show you the hds.

A

A

I've got it wrong.

B

A

ah What's the right way to run that hcs diag what's up perry? um uh Actually, you know it's good, because I have this in my notes. I always I always forget how to run this command um it's over here, hcs diag. It gives you your your list of all your gives you a list of all the containers.

B

A

How do I run this.

A

A

Oh cool well, this guy's on pinterest! What's this okay here.

B

A

B

A

Maybe it's just list, maybe that's all it is. I don't know.

A

Yeah there it is it's just okay, it's just it's just hcs diagon list. That's all it is. So if you do that, you you get to see all of your all of your containers right. So this is kind of like it's kind of like doing like a lower level version of docker ps, it's kind of like doing a ctr, ps kind of for a cluster where you're running docker or whatever, um if you're used to running container d command line stuff but anyway, so those are like the two main commands. So now this is running.

A

You can see all these docker containers coming up. So these are like traditional current sort of state-of-the-art windows clusters. You've got amazon, you've got azure, um you know these are sas. You can spin them up from command line tools. They work, they spin up docker containers in the amazon cluster. We can look at it later. It does actually because I manually installed calico and I'll, show you how that.

A

A

A

Yeah here it is okay, so you can see there. So I manually installed calico on this cluster. So the way you do that is you you pull down. um You just pull down a couple of powershell scripts and what it does is. It starts calico as a service right. So um this is I'm I'm ssh, I'm running rdp'ed into a windows node here and then I'm gonna.

A

Run so there you go, so you can see that I can. I can run um get service, calico and I'll just see my cni provider running that way. So that's how you set it now and you can see there's like some scripts here and then the logs literally go in this logs directory right. So um if I cd in there, I could cat it and whatnot and I could see those see the calico logs for felix and for node and everything. So that's how um hey? What's up scott?

A

um That's how uh the that's, how the calico? um That's, how the calico provider, if you're running the latest one is you're gonna, set it up um on a on an on an ec2 cluster. Okay. So actually it looks like the test finished running. So in this case.

A

In this case, what you can see is well I get what this tells us is. We have a lot of work to do right um in the community.

A

We still, we still do need to make sure we get um to the point where we pass all the network policy tests, but we're, but we can see that calico does indeed enforce network policies on windows, which is cool right um so so, um and um we will see that we can see that again with andrea, uh possibly later as well, um but well, maybe not because our node that we're that I installed antron is kind of having other problems right now, but um or what's actually perry's cluster.

A

That's that's running it um and we're also trying some other stuff like running privileged containers, but anyway, so here's calico. um You can see that it passes some of the network policy tests here, but not um all of them. For example, when I try to go from namespace y to namespace x, I'm able to do that, even though I have a deny all rule so um yeah so anyways. The point is that that's where that's how you run the end to end test? That's the typical developer experience on windows.

A

I think, if you're, that that's what it looks like running the end tests and then tests and compiling them on windows and everything else just wanted to like jump through that. You know it's kind of messy, but I felt like nobody's really shown that before and I wanted to make sure people knew how to do some of this stuff because it took me a while to to learn it. So, okay, cool, um let's see.

A

Let's see what else do we have? uh Oh.

B

A

That you're right hcs.net exe list yep I've. I eventually figured it out, um okay cool, so next we've got um what do we got? Next? um We've got this sequence diagram, so I wanted to start off with with with just a really quick tour of the windows stuff. If anybody has any questions since we haven't really even gone into the architecture of it all um feel free to sort of just ask them now.

A

Otherwise what I'm going to do is um I'll open up ravi's diagram, ravi, um oh weird file, the tr ravi. We lost your diagram, I don't know what happened to it: okay! um Well, he had a good sequence diagram, um but ultimately, what we can we can find it. I can show you another one, ultimately, the way container d. um So so now we're gonna, that's all docker! So now we're gonna move over to looking at container d right um and so container d is different and one of the fundamental things that's been tricky about.

A

It really we'll tell you this um who's, our andrea um hero over here over at vmware. um He is uh one of the things that's tricky about it is. Is there isn't a there? Isn't a connection to the host I'm in container d you get one for free with docker, um and so there's you can't do like there's certain types of things like, for example, like running containers that don't have an actual pod network ip address. You can kind of do that um in docker.

A

You can't do that in container d, where everything is moving to container d. As you all know, so um we need to find workarounds for that and there's some real, interesting work. We've been diving into and we've been really spearheading about how to make it easier to do that kind of stuff in container d, but even more importantly, the real solution to all this is privileged containers on windows and as of like last night, we got a sneak peek of what that actually looks like, and so let's try to see.

A

So I can show you right now. I think it's probably best that I just show you all right now what that looks like so um so we don't lose too much time because we started a little behind. So folks give me some feedback on terminal size here.

A

Can everybody see my terminal.

A

Bigger, please: okay, bigger, still bigger, okay, I'm just gonna keep making it. I'm just gonna keep going. Okay, eric uh look at this how's that is, it said, cheek cell, is that yes, as in yes, bigger or yes keep going almost there. Okay!

A

Here we go here. We go thanks mark this is it we want to get this right. This is um okay cool. Here we go. So um let me close this thing. Let me get out of this okay, so let me make sure I'm in the right, coupe ctl.

A

Let me make sure I'm in perry's cluster perry. Thank you so much um for uh providing me with this infrastructure to do this on. I did not have mark stuff running on my cluster, but I will show you my cluster. In a little bit. um I have a cluster api based whoa. Sorry about that folks.

A

I have a cluster api cluster that I'm going to show some folks in a little bit, which is which is uses cluster api to spin up a container d on andrea um environment, but- and- and this actually does the same- um but this is a special one that has mark's uh perlers containers. So privileged containers on windows are the thing that would allow windows to really come to parity with, um with with linux. um It's a huge deal.

A

um There's there's you know, there's like a there's, there's cups about it, there's um privileged containers windows- if I just google, this um yeah like there's, there's this kept here and amber left microsoft that she was kind of working on this and now I guess mark and his team is taking it over, and um so this is this is this: this is ongoing stuff, there's a lot of a lot of stuff online about it. I'm gonna jump over and just show it to you.

A

So if I do coop ctl get pods dash a here, um we can see that we've got several prometheus containers running and if I go wide okay, if I go wide.

A

I'm going to have to go a little smaller just so I can see, because I went wide.

A

We can see that we have uh our monitoring. Containers are actually running in in our windows in our windows, environment and we can actually get prometheus metrics uh on those containers.

A

So, um if I go to, where is my.

A

uh 10 dot strange, it's like I lost it. Oh.

B

A

All right, I got scared, okay cool see, so so here I am in this cluster and um you could see that I am actually so this is these. Are these are windows nodes and you can see these are like windows, specific metrics. You can make this bigger for y'all, um and these these windows metrics are being scraped up um by uh prometheus by prometheus agents, on windows, node and um and so, and because they're running is privileged right. You would not be able to get these metrics otherwise right. um These are.

A

These are running in containers. These are not host processes so, and we can look at so our little joke our little inside joke. Does anybody want to guess what this was? What these three peaks were, what we did to make these three peaks go away. We have kind of like a little inside joke between mark and mark and perry about that.

A

um Okay, I'll tell you. This is windows defender, so we were running the same test that I just ran a little while ago that spin up those nine pods and do all that crosstalk and I suppose I think what happens uh security folks can update me on this, whether I'm right or not. But when you we, we had this cpu spike in and it almost brought the cluster down.

A

It turned out that the windows defender does not seem to like the way we do lots of probing in the kubernetes end-to-end tests, um or at least not in the way. We do it in that test um and so um turned off windows defender and the behavior seemed to get a lot better and, of course, at some point, the test kind of just completed.

A

So again we could run these same tests on this cluster as well, and I will show you again how to do that.

A

I think I do it up here. I don't know perei, I don't remember anymore. It's been a long week. Okay, here we go so again here, but I'm going to show you something else that I think is nice to keep in mind when you are diagnosing windows nodes. Is that the n10 tests? um Again, you know I already. I mentioned this node os distro flag that you give it, um but I also wanted to wait.

A

Can you see, I don't know if you can see this yeah claudio you're right, it's windows defender and mark somehow found a way to do the same emoji that he did in slack so anyway, this is all mark mark gave us the binaries to run this last night. So we really appreciate that mark so that we could start testing this. This is going to completely change the way the community does everything related to windows, um and so um I'm afraid to exec into this, though perry or mark do you can?

A

If I execute this container, is it going to crash my terminal? Let yesterday it kind of didn't it wasn't happening when I tried yeah, it's still a little web yeah, um but it's that's really cool so anyways on this thing, because perry always taints his nodes in windows. You have a lot of note, takes it's very important to think about taints on a windows cluster, because you don't want the scheduler to put linux pods on a windows node, and you don't want the scheduler to put windows pods on a linux node.

A

The images won't pull down. It's a total mess right um because remember your container runtime is kind of like aware of what it needs to pull down and it needs to. You know, run these processes and the scheduler is not um not natively, aware of like windows and linux, so so you've gotta, like have your taints set up and I'm gonna show you how runtime classes work in a second um but anyways. This is how you run this test.

A

Let me make sure now that I'm on my I'm on my, let me see what I've got in here. As far as the name space, oh gosh, here we go.

B

Okay, there we go.

A

Oh perry, did you clean that? Did you clean it out for me?

A

I think you cleaned it from me. I think that was really nice of you to do that. Okay, so we're going to run this those same network where'd, I put it.

A

um Let me go over here, uh yeah, I think I put it in here.

A

You all made me, do this big font, and now I don't know where anything is.

B

Anymore, okay: here we go here, we go here, we go here, we go here. We go.

A

E2E there we go so we're going to run this test and if you don't use non-blocking tests, if you don't, if you don't, if you don't well, I guess somehow it looks like we removed all the tanks. So fine, oh, I untainted these nodes anyways. So we don't need the blocking taints, um but anyways. Normally you use that flag so that um the n10 test will continue, even though there's like certain taints on your cluster. So this takes a while to spin up.

A

This is not going to spin up as quickly as it did on the other other stuff, we're using some some older hardware on this and and it's a much smaller cluster. um So but this is good because we can like just let this run and we can like sort of finally slow down and you've seen privileged containers. You've seen them running now on um you've seen them running on on on windows.

A

So that's the big news um and now we can kind of slow down and we can actually kind of look at some of the some of the actual like stuff that I kind of wanted to wanted to show you.

A

But I was afraid we wouldn't get to um this other stuff first, so um let's slow down so so the other big thing that's going on in windows, containers um and and again, if, if this is going a little too fast, please do like there's a lot of great talks, um especially from michael michael, a recent one on an introduction to windows, containers and stuff like that and there's a great one from deep um de bruy on an introduction to the csi proxy as well, which we're going to look at here.

A

But um so now we can kind of go into either the csi proxy, but maybe we can just do this. Do like a play at home thing. Do folks want to look at the csi proxy now or do you want to look at how cluster api works under the hood?

A

If, if, if I gave you a choice um because we can kind of go either way in terms of or do do you want to look at some other stuff.

A

Both claudio wants it all, okay, which order you want it in claudio. What do you want to do? First,.

A

Everybody wants to do both we'll definitely do both I'm going to pick one unless somebody tells me cluster api. Okay, all right fanatic, yeah,.

B

I'm not a cluster api fanatic.

A

I work at vmware, but that doesn't mean I'm a clustering, yeah, okay, um yeah all right. Let's do cappy, okay,.

B

A

Did so perry um has worked really hard on getting the image builder to um work for clustering and um james at microsoft actually started that work, um and uh so uh the image builder um is kind of the the secret sauce behind how cluster api works with um work works in general, and it's one of the reasons why it works so cleanly with windows.

A

So um is that you know it turns out it's real interesting, like you know, when we first started doing this stuff, I was like building ci jobs to like run, custom, containers and stuff, and in the end it was real weird.

A

We just realized that we didn't even need any new custom, tooling or any binaries or any weird stuff in order to run windows, container d stuff on cluster api, and so I was just kind of talking to nadir about this the other day and some other folks um that are on this call and about like what was the deal why? Why did it all turn out this way?

A

And it's just because of image builder, really the the fact that you have these immutable images in cluster api or the fact that you just drive everything from the image gives you this really modular orchestration framework for spinning up um spinning up different types of clusters, even clusters that have multiple different operating systems and stuff. So if you really look at it this yaml file, I know we, I know everybody doesn't like the ammo, but so this is what the way I spin these clusters up and the way perry does.

A

It too is we um we use tkg, which is the tonzu. You know cluster api based um like downstream product, but you could do this with upstream as well. In fact, gab who did all the csi work who's here will tell you he does everything using the upstream cluster api tooling, and um what we do is we we we just have a. We create a yaml file that defines that sort of redefines all the basic cluster templates and there's almost no custom logic in it.

A

Right you've got a cluster, it's got a cider block right, that's nothing! New right you've got a vsphere cluster, which is just an inherited thing from that you've got cpu you've got memory. You.

B

A

Need eight gigs of memory. You can't do four gigs of memory. This isn't linux so for these windows notes, you've got to have eight gigs in memory. um I learned that the hard way- um and uh so uh you insert your ssh keys, you've got a machine deployment and that's kind of really where the customization is right, because so a machine deployment is a cluster api concept of like a kubernetes pod deployment.

A

We've probably done this before um you've, probably seen some of this before in um in other tgik talks right um and a machine deployment just creates new machines so like.

A

um If I get out of here, I can say: coop ctl get um well, actually I'm not on that cluster. So I can take coup. Ctl get nodes right, and so you can see. So this is a machine deployment that has a size of one right, so um I could say tkg get uh clusters, I can say tkg get credentials uh or I can see tkg get management cluster okay, so I can get that and then I can um use context.

A

I can do uh coop ctl group ctl get credentials, so I could, if I want to use tkg. If I want to use this cluster, I can ask tkg to get those credentials and then it'll give me this command and I can switch to it, but I can also copy paste this command and I can use this and I can take this command bear with me here. I can take this command.

A

A

Where is it, why did I lose my um yeah here.

B

A

Is group ctl and I'm going to my management cluster? Is here uh yeah this one right, so I can use this. I can use this cluster right and now I can say, coup ctl get clusters right and you can see that I I just have uh um so so it's these are crds right. So cluster api has these cluster crds and um it has all these other crds in it. So this is now I'm using my management cluster to manage my kubernetes cluster right and anyways. I can do things like.

A

I could scale deployments up and down. I don't I've never tried this with the windows. Node, I'm going to do this right now, perry, tkg, scale, tkg scale, cluster um tkg get clusters tkg scale cluster.

A

I could grab this dash dash. um I think it's w2.

A

um I don't know what that's doing, but anyways, maybe that'll work. Maybe it won't. I've never actually tried on these clusters to scale things up, but um so that that's how so that's how all this works- okay, it actually did yeah it actually is, is working so so it's actually creating a new node now so um going back to that yaml file where's my yeah, going back to that yaml file. So let's see what this when I did that when I created that uh cluster definition, what is that machine deployment? That's being scaled there right?

A

So if I go into this deployment, this is where the windows stuff comes in right, so you have a kube, adm, config template and actually what the magic magical idea. This is uh perez plumbing that he figured out for getting this stuff working, because this is container d and then we're running android, which is our own cni provider on container d. um And actually, if you look at the installation process for calico, it's very similar to this right. You basically are w getting stuff from from the uh from the internet.

A

I just I just saw yeah um by the way uh yeah james um james announced today that some has some has some great announcements about all this stuff um and so big shout out to james by the way for really bootstrapping all of this stuff um on on windows um from the cluster api perspective. This is huge, so um this here is how you actually install android as a cni on um on windows in container d. So what what we're doing here is?

A

We are pulling down an installer for open v, switch okay and that open v switch installer um is then uh it's an unsigned binary? It's just! You know this isn't like super secure or something, but whatever um you know, it's just for testing right, um we'll find ways to productize this everyone, I'm sure we'll find ways to productize this properly, and this is for upstream um turns off the firewalls right um and uh starts a user space coupe proxy.

A

So andrea runs a user space coupe proxy, so as not to collide with uh hns, um because it uses open vswitch instead of a kernel space um user proxy, and then it um starts the android agent up as an nssm. So nssm is like a windows, service manager, systemd type thing right, so you fire up this uh native, like windows, service and um and that's intrigue. That's how you're running entry now you do the same thing essentially for calico right.

A

So if you go to the calico docks- and I have that in here right so if you go over to the calico docks, oh gab, I see your name down here. Gab got csi provider working for vsphere, which is really cool on top of csi proxy, and I really want to show his work off, but I just can't stop talking about andrea hold on. Let me see uh where is it.

A

Where was it, where is it the calicoin installation instructions yeah? Here we go so alex. This is a cto over it. Tajira sent me this.

A

I I think I asked him on twitter and and then he he and then he told me how to how to do this, because I wasn't really sure what the right way to install calico on windows was, but they have a blog post about this um and yeah here it is so yeah if you look at what they actually do and they explain it here right, um oh gosh, I just pasted an entire.

A

I just pasted an entire shell script to ins. I tasted powershell into the term. I just pasted powershell into my browser window. I don't know, I don't know what that's gonna do to me. Okay, so here we go so so you can see here. Basically these are you see these pre-bootstrap commands right, so this is like kind of like this is kind of the same thing that we're doing with andrea right I mean there's some bootstrap commands that are running on the node. These are eks specific, using their little eks dsl.

A

To do this, but again this is one of the great things I think you know kind of one of the cool things about cluster api right is that you have kind of a kuba we're using kube adm directly um and we're modifying the bootstrapper directly now nadir, who I think nate here is here, but it's late in england. I'm not sure. If you talk to him, there's going to be something even better than kuba adm bootstrap. Eventually um that comes in the upstream kubernetes upstream cluster api world.

A

um That's going to allow us to do this stuff much more declaratively, but you can see they're bootstrapping the same way here watch out. If you run these instructions you can you? Can you don't want to pull down the darwin kubectl? I remember messing this up, so if you do run these upstream uh calico installations on eks you're gonna obviously um want to pull down the um the windows kubectal right.

A

So you do this and then you literally run, um then you you can you and then you literally run this um this powershell script, and so, as you can see, this is not that different than what we're doing to set up andrea right so where's, my terminal right.

A

So we're basically doing something very similar here right, we're we're pulling some stuff down from the internet and then we're installing andrea, we're installing it as a service um here and here right so um yeah, here's, the and here's the um here's where we actually do the entry uh powershell installation. So this is a kube adm thing right.

A

So um once once kubedian bootstraps it does all this andrea stuff and then it bootstraps those windows services and it installs, ovs and then you've got um you've, got the open v switch stuff running and you've got the ip address. Space is being allocated at the node level um via ipam for you through andrea. So so that's the way that cni works now. Okay.

A

So I'm just going to look at these questions because I haven't yet sorry sorry mark um it's an unsigned driver, yeah, um okay, so james post-pre cubanium won't be needed once we have privileged containers exactly so that's what's so cool about the fact that mark gave us those privileged containers. Yesterday, it's like we've already got him working with prometheus.

A

So it's like you know, they're not going to be mark actually asked me to mention that these, like not to jump the gun and calling these privileged containers. These things that I just showed you because they're not really privileged containers. We don't know exactly what privilege containers are as far as I'm concerned.

A

They just feel like privileged containers, so whatever and pretty soon andrea will not be using the user space coupe proxy, so andrea has its own proxy, and so the only reason, but one thing- that's interesting when you're running a windows cni um both calico and android do like- and I haven't fully dug into this, but they both use coop proxy. um They both kind of have their own like setup of coupe proxy.

B

A

Kind of have to have a higher touch sort of interaction with the cube proxy. We kind of just take it for granted on linux clusters because it works so cleanly, but on the windows clusters, you got to think a little bit more about it. um You know whether you're using it running like, for example, in in calico, it runs in in kernel. Space in in in android runs in user space.

A

um If the aks folks I'd be curious to know I don't know, maybe we I I mean I don't know how you all run coupe proxy. I assume you kernel space in aks. You know I'd be curious, though um kubectil actually did you're talking about kubectol, okay, um so yeah. uh What else do we got here? So um now? Let's go uh what else so we have okay, so we have the instructions for andrea. I think I went too far down.

A

um Oh yeah, I didn't even get to talk at all about windows, images and stuff, like that. So um so I'll show you something that, if you're new to windows you need to know, um and so when you run windows, images, there's kind of like you know, there's all these different variants right of for any given image. Now, there's multi-arc, builds um and, and so like. The container runtime can be smart enough to like pick the image version.

A

um If you have a multi-arch image, but if you don't have a multi-arch image, you do have to tell your windows node, what type of image to run and there's a few ways that we have in kubernetes to make this easier to deal with. One is runtime classes.

A

And let's see uh do I have an example. I thought I had a good example in here. um Maybe the best thing to do is to just show you on a running cluster, so.

A

Okay, so let's go back to this cluster here these tests are still running. This is a really weird cluster. You know it only has one note on it and it's being scaled. I don't know if it's even working anymore. um So if I go into here, tgik right, um if I look at this iis pod 2. yaml right.

A

So if I look in here, um you see this runtime class, so this is real important right, and so this runtime class is something that you create in your cluster and you tell it the container runtime that you need to use. So, for example, if I do coop ctl get runtime class.

A

Actually it needs to be runtime class name. I don't know why. I called it that it's yeah, it is runtime class name. I got it sorry, so so you can see here. I have a runtime class right and um wait. That's not the runtime class yeah here it is so cube, ctl edit runtime class.

A

So this this I had to figure this. I figured this out on christmas. Eve. Perry wasn't around to help me, so I figured it out by myself, um and so, where is it it's um here? You've got to get this handler right. You got to get that handler now so you're asking so you're asking now. What is this handler so what this runtime class does?

A

Is it says that if somebody says I want to run a windows container, if they say the name of their runtime class is windows, then I'm going to like give it a node selector and I'm going to give it a toleration right. So this is the idiom that that that allows you to sort of declaratively schedule, windows um containers you might need to have multiple runtime classes.

A

If you have multiple different types of windows nodes or on your on your cluster, which you may very well have, because certain windows containers won't run on certain windows nodes and that's not a trivial thing in windows. It is in linux for the most part.

A

So now this run hcs wcal, proc, wclw, process, process uh process thing right. If you go here, dims showed me this tool. I love it grep.app, it's my favorite! You can look for code anywhere on the internet, and so um so this this. uh This is actually a default of the windows container d right. So if you look in here, um I know there's a little latency here.

A

So if you look into the windows um container decode that's used to launch the hcs stuff, um you you, you have a default container, runtime called run hcs wcal process and then that gets sent by the kubelet. um That could that's a that's an argument that gets sent from the kublet to the underlying um cri right.

A

So the way this all works is container d first um goes and it talks to your cni provider, and it says I you know I need a network, your cni provider returns and then, after that you know, then it starts this process up and it's going to use this default runtime. So this needs to match up with what you've got in your runtime class here. Okay, um so uh don't screw that up? Okay! So now, um if you don't have a runtime class, then it's not really it's not the end of the world right.

A

If you don't have a runtime class, then you can just manually put a node selector on your windows pods, so that they always run on a windows operating system right um and you can manually put the tolerations on there or if you want you can just you know you you can do you can do you? Can you can just do all this stuff manually if you need to and um to get the end to end tests running for kubernetes?

A

That's exactly what we're doing we're manually, labeling things, but then ravi um red hat ravi told me he wants to move towards runtime classes the other day. So I don't know maybe we'll all move to runtime classes at some point. But for now I feel like the n10 tests. What we do is well I'll. Just show you how we do it. We we just manually label these things and go right, so we literally go in here and we just grab the pod.

A

We put a notes, electron at windows and, um and so that makes these run. That makes it so that these are likely to more likely to run on different types of clusters, because earlier I showed you eks and aks, they have different conventions in terms of what they label and what they taint and what they don't tame right. There's no standard for windows nodes. Yet so this is a but one standard is that you do get this label on every on every windows, node right, so good scheduling trick! You can use okay.

A

So, um by the way I'm wondering ravi, if you're watching have you run the descheduler on a windows node. Yet because I'm just curious whether like whether people are using those on on windows clusters, because we found some like process leaks and stuff recently um yeah claudio, so you can yeah, you can taint the mask yeah!

A

Well, the master's usually and the master is usually tainted right. So um I'm looking to see if there's anything else, okay, cool! So no other questions all right. We can keep going everybody so um but ravi. I am curious about the d scheduler stuff um and um so uh okay cool. It works interesting. um So uh I I the reason why it's important, I think is I I just have a feeling from a little bit of experience, not not a whole lot of large scale testing. I have a feeling.

A

We're gonna need a lot of defrag stuff going into the windows stuff over time like to make it work at large scales. um So.

A

These commands um get processed. I think we've been through that um for folks again coming from a linux environment, which is one of the things I wanted to show people because that's like where I came from, and I didn't know how to do any of this stuff. um You you have to here's, how you install vim through the terminal you in in windows, there's a terminal tool called chocolaty, so you do this, it installs vim uh uh and then you know our installs a package manager similar to homebrew.

A

Then you do choco install vim and then you can do you know, vi and editing stuff. um You you, you don't have a docker demon anymore. I mean most people, don't um so it's more important nowadays than ever to be able to run uh ctr to to to see what's going on at a low level with your containers, ctr is the container d runtime uh client. It comes on your uh container d nodes so um and remember.

A

If you want to list all the containers you have to do dash dash name, space equals k8, dot io, because container d containers have name spaces right. So that's just something to keep in mind. Okay um um and those name spaces aren't the same as the kubernetes name spaces that you're used to thinking about right. um So uh I don't know why these updates keep going. Okay, um cool, um oh yeah, and just one other thing, if you ever do- need to run a windows container and find in that hub.docker.com over here.

A

If you need to line up your windows os with um like, if you have a windows container and it's not running on a host- and you don't know why you know you can go here and you can look up all the operating system versions for um various windows containers and then find the docker tag for it. And then you can run this like in powershell. So I think I've got a windows note over here. So if I do ctl get nodes.

A

Oh I'm in the wrong use.

B

A

If I do coup ctl get cluster tkg get clusters. Switching back, I should have installed coupe ctx on this um tkg get credentials going to go back into my other cluster. So I can show you this okay and.

A

Oops now, if I do um group ctl get knows I'll, just jump into it to a powershell terminal on one of these.

A

Wait am I in the wrong.

A

That's weird: uh what happened to my windows.

B

Now I think I killed my.

A

Weird, I don't see any windows nodes.

A

That's odd, um I somehow managed to kill. uh Well, luckily, I've got I've got another cluster. So uh let's see here, coupe ctl.

A

A

Yeah, I don't know what I did.

A

I it's like. um It's like magic, it's gone. um So if I.

A

If I want to, um if I do want to run those ctr commands, though um I don't know whether scaling failed, but it seems like um I'm really curious, I think maybe we should just dig into this. So if I was to if I was to go back to using my management cluster again right so um now, I've got these in my history thanks to control r history, correct what are the contexts here yeah.

A

So if I go back to my management, cluster, okay and then I go and I group ctl get nodes that looks right and then I go, and I and I create cube ctl, create dash f.

B

A

I'm just deleting that whole cluster and then I'm going to recreate it. I don't know what happened. Something weird happened on this box. It's like I deleted something on accident.

A

Let's really clean it all out: let's let that give that controller a second to breathe. um Meanwhile, we I'm just going to do a sleep 60 and here's what I'm going to do! Here's, what I'm going to do for y'all, I'm going to do a sleep, we're going to sleep for say how long should we sleep? Let's say 20 minutes and then 20 seconds, we're going to sleep 20 seconds and then we'll come back.

A

A

There we go okay, so let this come back, um it's going to recreate all the infrastructure and everything else, um just by the way, it's kind of interesting to see how all this stuff works. um This is um these clusters are running on. I I spun these up on. These are on nsx and um for folks that um are just wondering like what, like a production network for a cluster api cluster, might look like well so here you can see cluster api work.

A

You can see all these nodes being created. Now um we've got a windows container denote.

A

We've got these three, so that's what cluster api is doing under the hood right for us, and um so so we can see all this stuff coming up and if, if you, um if you look in here so the way, the way like at vmware that we do with this stuff right is we have these um for testing on nsx, which a lot of you know, people running in production, want um we have these t1 and t2 topologies and um that what they do is they set up uh like a network plane for all the vms right?

A

So that's how this stuff works here and this stuff is all in terraform and like if you wanted, you could play with it. It's on github, um this really nice guy named dumloo, um uh helped help helped help. Helped helped me to put all this some of this stuff together and uh and also nishad, of course.

A

um So uh that's how that's the network for these that's the way this is working, so we've set up an nsx network, the software defined network, and then we've got a vsphere instance and that's like where all this infrastructure is running, and so you can see cluster api. My management nodes are here and then here's my worker cluster, that's creating it's creating two, um it's creating two container d nodes and it's creating two uh like linux, nodes, right and uh so anyways that, while that runs, we can now how much time do we have?

A

Oh gosh, it's already 5 18., so yeah doom-loo big up to dumloo right yeah, doomwoo he's my friend, you know what let's go, let's find doomlow on the internet. Where is he? Where is doomloo, I'm going to find his github um joe or anybody else? I don't know what the time line for these usually is. I don't know if I should keep going or not. I don't I mean I have no idea. um I see that um I see that we've been going for uh about an hour and 10 minutes.

A

um I don't know if I can find it but anyways since, since you know doomloo, um if you do have his um his get up, I'd love to link people to that terraform recipe, um cool thanks, scott, okay, um so um csi csi.

A

Let's look at what's going on in windows with csi, okay, so deep really is a really nice guy and he made this um csi proxy windows, so he made him and a few other people, I'm not even really sure who did it um made this thing, it's the um and I got them to merge it for me. Yesterday I used this ggki as a as an excuse.

A

I was like merge my version of your diagram, so uh deep made this diagram and I ported it to ascii doc, and um this explains a really interesting problem that we have, because we don't have privileged containers up until last night at around eight o'clock, when mark rosetti sent me the privileged container, but um up until now we don't really have privileged containers in windows.

A

So how do you do things like csi and cni right um so you've seen all the low-level details, so we're just going to take a take a breather and like step back a little okay. So how does this all really work right?

A

Well, so what we're doing is, as you saw earlier, we're running those we're running andrea on the nodes right, we're running calico on the nodes, we're running these things totally, not idiomatic to the way you would do it in linux, right where you know you always run the cni provider as a container, and then it just you just poke holes um to talk through ports to set up firewall to set up the um you know, iptables rules and all that as a privileged container right um or to talk to ovs or to add network devices to processes or all the other things you need to do.

A

You can't do that in windows, um so the csi proxy and another tool called rancher, wins wins which is kind of notorious and famous. Now, for being like the glue that we use um in in in various windows scenarios, that gives you basically a proxy like a back door that allows you to basically run commands on the host from a container.

A

So if you run andrea right now on windows today, like on a docker node, what you actually wind up getting is you have an android container that it's not really a container? It's just a thing that um sig windows we have here, um sig windows tools, um hey mike, we miss you, man mike left us. He went to go, do something else, but um anyways.

A

A

You can see in here right. um What really happens is you you, you have these like sort of these powershell scripts and the powershell scripts, basically sort of configure um some stuff to like talk to the outside world and to to go into wins to to basically run a client and what that client does. Is it talks through us proxy through a socket to a thing, that's running on the windows host, okay and um that's how you do the equivalent of a privileged container right?

A

So it's like you're, you know like you're, you know it's like you're, um you're, a criminal and you're. You know on the inside and you're like calling your your your friends on the outside to do like the dirty work, kind of thing right, that's kind of what this whole thing is, so it's really cool that we have these tools that do this, but so rancher wins allows you to do anything. You could run any container on the outside world. Csi proxy constrains that right, so that you can only run these csi specific commands.

A

Okay and so gab got this working for vsphere. Let's see if we can find gabs commit uh yeah. Where are you there? He is there's my friend there. He.

B

A

Okay, so gab wrote this and what this does is. It allows us to um like for vsphere um to work with the csi proxy and what the csi proxy does is it's a thing that does kind of the same thing that wins does, but it does it in a way. That's structured and sort of enterprise ready like it doesn't allow you to just run any arbitrary command right it it kind of it um hey. Do you all use git pod?

A

It's like my secret thing that I like to do. Is fire up get pod on like random github repos? um So we can. I love it because you can just click this button and then you can just grep around and find stuff, but anyways. So csi proxy wraps these commands and um it only runs certain commands for you, so you can't just run any command. You want right. So if I go in here, get gret ps1.

A

Yeah, so you see all these like internal like well yeah like see all these like internal. This is like this is, I know you all are gonna get mad at me. This is too small. Here we go here. We go there we go. Can you see that mark okay?

A

Is that visible to y'all? Okay? I think it should be. If not, let me know okay, so so what it does is. It runs the commands for you right. So it has all these like little wrappers to all the to the to the um to the to the underlying windows system, calls and stuff that you need to make to like mount disks and mount mount volumes and stuff.

A

So it's what allows you to run csi on windows right like to be able to run csi provider on windows now gab has uh gaba has got a url that I put in here. I I gab. I just I don't think we're gonna have time to do it, but we should just we should just wing it and just see what happens so. um He's got a url in here where he shows how to get this stuff. Working on vsphere, um let me see where is it? Where is it? Where is gab, where.

B

A

A

B

It is yeah, so he's.

A

Got instructions on how to run the csi provider um so that it works with vsphere, which means you can run use the vsphere csi driver on it, and so I think this cluster already has a csi driver. um Wait hold on yeah. This is it: where is the vsphere windows prototype here? It is.

A

This fork contains a windows prototype until privileged containers are supported on windows, it's already obsolete gab, because because because we have them now so setup of vsphere cluster with csi and windows nodes ensure csi's proxy is running on each so to install csi, it's just a daemon set right. um I don't think that's particularly hard, um so wait. What is this.

A

Yeah here it is so yeah I can copy paste this daemon set and I think that will create.

A

I don't know we'll see we can try to at least get through three. I'm gonna I'm going to wrap up in a second here, but we can at least.

A

Oh, I know what happened. I know why I lost my nose. It was because.

A

A

Can I just use that daemon set as the csi proxy damon set gap and then oh gosh? I can't.

A

Csi proxy runs a damon set right and then the vsphere is a different one. Right vsphere is the other storage damon says so it seems like I was thinking. I could just run it and then run the other one on top, but we only have like two minutes left. So it's probably no point but yes diamond.

A

um I don't even think I have nodes up, because you know what happens I mixed upstream with downstream, which is a bad idea. So I think I'm I I think tkg did something with those nodes, because we were running all this weird upstream stuff and we that's not integrated into the tkg thing yet.

A

Let's see what happens here.

A

I don't really expect anything good to happen, but I'm just kind of curious.

A

A

That csi proxy that daemon set that starts so so does that mean that we need to run csi proxy gab when we run it, we need to run it as services on the host. Or can we run it like? We just have to run them as oh yeah. We have to run them as executables on the host. I guess because otherwise, how else are you gonna? Because we don't have privileged container containers so yeah?

A

um I could parry, but it's like we've got two minutes left, so I feel like we should just go breath first and get through the other stuff um so um but before yeah. So this is this is the this. Is the thing that I kind of skipped over right see so sc.exe create csi proxy.

A

uh So this is, I guess, creating a windows service for it. So that's really what we do is we run this on every host, um yeah and then and then so we'd exact into them.

A

So, and we we also have to would have to download it next time. Maybe so um I just want to make sure I didn't miss anything else that was really really substantial for people to understand log file, locations, everything's in c var log I mean it seems obvious but like for me, I took me like a week to figure that out so um for windows, um and then we've got all these pr's. These will be uploaded with the episode.

A

um One other thing is this ongoing container d issue, which I wanted to talk about. um We, this is probably the most interesting thing. That's going on right now is, um I think, for from our perspective, from a networking perspective is the fact that um we really want to find a way for container d to work cleanly with our cni's, um and so probably so. The problem is that right now um I mentioned earlier. The the code path is that cri makes a call to uh when you start a container.

A

The kubelet tells container tells the cri interface on the contain, on top of container d, to run something, and then that ultimately calls down to um to your lower level container. Runtime, which um calls which, which calls the cni to get an ip address uh to get the ip the network set up the ip address set up, but then in windows.

A

You have to then, after that, um like there's a step after that, where vnick's created um and that's because of the logic of how hcshim and how the windows internals work, and it turns out that that is causing um that's making it so that we return ip addresses as cni providers. When you, you might return an ip address that isn't ready, yet isn't routable yet may not have even been uh plugged into the container process.

A

Yet so um because of that, uh we have this weird kind of race kind of guess: work condition that we do as a workaround in andrea, and this is also affect uh calico as well. So they've root caused some issues with the calico on continuity as well to this, and so this is something interested that we're looking at. If anybody knows about the cni spec, the cni spec has a check in its api call. Now I'm not super familiar with what you can do with this, but antonin over from the andrea world.

A

Yesterday was proposing that maybe we can use the check to get rid of this race condition um or not get rid of it, but like kind of like ameliorate, the problems with it, because we can like use that check to continuously make sure that a network is up. And if the network isn't up flag. The pod as unhealthy. Somehow.

A

So that is the hottest thing in container networking for windows that I know of right now. um So I know a lot of this was about windows. I see your question eric about gmsa.

A

um You know who I don't know who one of us either me or or mark, or james or ravi, or someone who who's kind of involved in the upstream windows stuff can answer that I haven't been tracking it, though, um according to perry, it is scheduled for 1.21, so come on into slack and um and hang out with us.

A

um If anybody wants to do another one of these hack sessions later, this has been a a little tricky to get all this stuff working for me because it's my first time to do a stream, but um and and learn it in a little bit more methodical, sort of way um like yeah, like let me know like we can just like do this offline sometime so um james says we're waiting for windows update and once that lands. The container gmsa stuff should work um yeah, oh perry. We need to start working on gmsa.

A

huh Well, I guess gab is actually going to be working on that. Actually pretty soon, so I think he said I mean. I think this is a good place to stop, because we're kind of at the one and a half hour mark um for folks just like reach out and see windows.

A

You know, I'm junior, 100, upstream slack, perry thompson is um perry. Thompson is a great resource to mark and james who are both from microsoft are also here at mark rossetti, james sturovant, um and uh so we've got people from vmware involved. We've got microsoft. Folks, we've got folks at red hat ravi's involved heavily helping us with test grid on the windows stuff and he's worked on the kubelet and stuff like that. So there's a lot of people in here that are really friendly. That can help folks to get more.

B

A

With playing around with like kubernetes workloads on windows, I'm also happy to help if somebody just wants to set up a development environment for upstream kubernetes on windows, or anything like that. um So yeah uh thanks thanks, keith yeah, um okay, cool.

A

Yeah thanks everybody for coming, so, let's um so I think that's it! Unless anyone has questions, I think I'm gonna close the stream. I don't really know how to close this stream, but I I see this big red button that says end stream and I think I'm gonna, oh, and if I look at it it has all these graphs and it looks like the everybody left. It's like there's a big drop off there's only like 40 50 people left, okay cool.

A

um So one two thanks: everybody thanks all everybody, james arvind, oh hey, arvind, how you doing um it's good to see you here! I I I forgot to mention that you know arvin's working on the openshift stuff with uh ravi and that crew over there, and uh so a lot of people are working really hard on this, so see see y'all later and thanks.

A