Antrea LIVE !!!, 27 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Episode 26: apiserver stress testing daemonsets, CRD statuses

Description

In this episode we'll look at ways to stress test k8s clusters and, maybe, see if we can reproduce any interesting etcd race conditions... like https://github.com/kubernetes/kubernetes/issues/65517 .... (and https://github.com/kubernetes/kubernetes/issues/109399). Maybe this will result in a new Kubernetes e2e ? or a cool test we can run on the side! BTW , did you know theres a little bit of a race for CRD creation and CRD request availability in the apiserver? Maybe we can find out why !

A

Okay- let's see so I'm gonna see if we can.

A

Here we go, I'm gonna see if we can edit, um if we can um create like a race like a race condition, type thing um today, so I've coop ctl get I've. Let me see here, docker ps, um okay, so I just created a welcome to andre life. uh So it is. um I kind of just did this show last minute and I don't even I think I announced it yesterday really late on twitter. So I don't I'm not sure how many people are gonna show up, but all right.

A

Let's do I have got I've got our k8s prototypes kind cluster up and um what I'm gonna do is I'm going to um I'm going to attempt to nikita showed me yesterday. So I saw yesterday that, if I did, um if I did, where is it I'm going to look it up in the show? If I did uh there's a there's, an issue where it with crds, where, if you.

A

Is it um see, I think if I go to here beautiful playlist, I actually put it in the show notes yeah um here and if I go well, let me open this up right. If I go to.

A

Right here right that getting the new crd name and then there's another issue that I'm going to try to look at, um which is this one right. um This is one I filed and I think mustafa might join us, but I kind of sent him the link last minute. um So I don't know we filed this issue about a week and a half ago or two weeks ago about we needed this disruptive test for upstream. So maybe something that pushes the api server beyond what you normally the limits. You'd normally push it to.

A

That would then expose the issue we talked about about a week ago, where we have this issue in etcd right where.

A

Where you know, 3.5 up to 0.2 is not recommended for production. Now that has been fixed and there's a new version of that cd out at cd there's a new version that's out um and let's see where the release is right.

A

So this is out 3.5.4 is out. 3.53 is out whatever um critical correctness issue. uh Oh, this is a different one. um I think I don't think we had talked about yet.

A

um Was that the the core issue that was filed- I don't I don't know- was it called the critical corruptness issue? I don't know anyways actually, two days ago, they just uh yeah 3.5.3 and 3.4 the data, inconsistency idris, who is addressed right so that this issue came out and it's the next two point releases of etc fixed. This I know vmware tons was going to have this soon and uh we're going to soon put this into our next release and I'm sure all the other k-8s providers are going to do the same thing.

A

So so thank you to everybody in the etcd community, hey vivek, what's going on, uh so thank you for to the etcd community for.

A

3.5.3 plus right, so that was a big win and um you saved us for whatever I wasn't able to reproduce this issue, though right. So we looked at some of this um some of these things last week and and how you can do the fcd ctl perf check stuff. um But now so, let's see if I make a cluster, so I have a cluster now and if I say coupe ctl get nodes, I've got a three.

A

uh I thought I meant to make three control planes, but it looks like.

A

um uh Coop ct okay, so I'm gonna do kind, delete cluster right. There's a cluster in there called kind kind, get clusters kind, delete cluster dash, dash name, equals calico kind, delete cluster dash, dash name equals entry, I'm going to delete both of these, um and so I'm gonna create a new cluster and kind get clusters.

A

Docker, ps, okay, there's nothing doc or ps. Yeah! There's nothing in here all right. So I think so. I I just sort of hacked this up. uh I figured we'd do calico this time because we haven't done it in a while check if it still works um but anyways. So um if I I found something yesterday wherein um if I.

A

A

T-Box a-t andrea, live okay, um where, if I, if I go- and I continuously create the the same crd like over and over and over again right um and then I check whether it's up and running like right, if I, if I do coup ctl, get that crd every once in a while, I found that I could um get into a state where for split second, there would be a um there would be a like a split second, where the api server didn't know that that crd existed, and I kind of thought that was interesting when I saw it yesterday, because I didn't know that was the case.

A

I didn't know that there was it.

A

I didn't know that you could create an instance of something in kubernetes and then that instance would would not be like you could create a new crd type and then it would say that resource type doesn't exist yet, and it turns out that I think there's a some kind of a status for crds.

A

So nikita showed me this issue yesterday right um so so she showed me that I she thought this was probably the issue that I was running into um and it looks like people are still asking if folks are sort of have have been able to fix this, and so I thought it might be a nice little quick, interesting thing to go through and see if we could reproduce this and then um see see if we could read through this issue together and see where it's coming from so uh all right.

A

So here's a little more info and how I'm so.

A

Actually, I guess the good news is that there's actually a more maybe there's a stable reproducer that we can use here because he's creating this so he's creating this crd in his cluster, uh who did this brian brian made this crd and then, after brian makes this crd.

A

He's saying this, this is not in and of itself. I don't think a very obvious reproducer, because um I had to do this hundreds of times in order to, um but maybe it's just that coop ctl applies fast enough right that this works. So let's try this all right. So, let's see what happens so if I say, let's see if my cluster is up, ctl get nodes.

A

Okay, so I've got two workers and two two workers and one.

A

I kind of thought that I had multiple workers in this.

A

A

Okay, if I go in here, um I think I meant to have multiple of these, but somehow I only have one.

A

Plane and then I'll do roll control, plane right and let me see where I vim local up yeah here. It is it's because we do it in here. So let me fix this, so I'm gonna do uh roll control plane, so I'm gonna make multiple control plane nodes here right. So if you're doing this in kind, you can make multiple control plane nodes and I think we only need one worker. I don't think we need like a million workers right. So if folks want to use this recipe, it's not like it's.

A

You can just clone this kind down and you can run kind local up and you give it the environment variable or the cni. You want and it'll create this for you so um so anyways. Where do I do it? Cluster cluster equals calico ricardo, wait.

A

Why are you being so nice to me today? How's it going? um I love you too, um so so here we go. If I damn it ricardo all right, I'm making a new one. Okay!

A

So once this comes up, then I'm gonna try to reproduce this and maybe we'll see if we can find this figure out what the situation is with this race condition- and I don't know how far I'm going to get in just one hour, making this stamen set, because I'm I'm hoping to make an ap a daemon set that can spam the api server.

A

But I I really didn't have any time to prepare for today, so we're just gonna be like we're: gonna, be kind of winging it more than more than usual right, um okay, so this is hopefully coming up now joining more control, plane, nodes, okay! So let's wait for the control plane notes to go scott's here, scott!

A

Do you, okay, scott? So I want to make a damon set and I want the damon set to spam at cd and I haven't really thought of how to do that. Yet I mean I have some ideas, but um do you have any ideas? So I I I filed this issue and I filed this issue upstream and I haven't gotten a chance to look on it yet look at it yet. So I thought maybe we could. Maybe we could try to do this today, but the idea would be. Let me see the original um yeah.

A

I have it, so this was the original idea. I I don't know if this is any good, though, like you have a damon said: you're continuously just updating annotations or something like that. I mean that's going to hit that cd right and then um continuously pull it for changes and then, if you can ever detect it a difference between the nodes, you know that that's kind of what I'm wondering, and so I don't know how far we'll get so, but that's my.

A

That's my plan, we'll see I'll see whether we can at least maybe get some pseudo code for that sketched out. um I don't know okay yeah here it is.

A

A

This is coming up now.

A

Okay, my cluster is up and.

A

Let's see if I have multiple nodes, um okay, the cluster is up so now I want to see how many do I have ncd running everywhere. Do I have multiple lcds or one- I don't know, um yeah? Okay, we have multiple, ed cds, so that's good more more chance for things to go wrong right uh either that are created and delete, random config maps and secrets aggressively out.

A

Yeah, so um so the first one I thought so, in other words the yeah. The overall theme for this is like race conditions at the api server level right, so the first one will be the one that I saw yesterday um and so the one I saw yesterday was this right.

A

What was it um yeah, okay, brian pursley? Maybe we can get him to.

A

Come hang out with us sometime, um okay, so wait! Let me check so my mic. I think my microphone just went off because I saw I stopped hearing myself. Oh hold on testing testing testing yeah, I'm still there. Okay.

A

There we go now. I can hear myself again: okay, um yeah, okay, so.

A

So, let's try this this this this guy's idea um so he's going to do this and then right afterwards, let's make a shell script. That does this right, so we'll call this vmcrd race, dot, sh and I kind of want to see why this happens. um So, in order for this to happen, you have to do coop ctl get foos get foo, coupe ctl get foo, coop ctl get uh jesus, coop ctl get foose right, so echo three echo two and echo one right.

A

So, let's see here so actually I'm not going to echo one, because I want this to be so fast. I want this to be like all right.

A

Here we go okay,.

A

Who's tapping: do you hear what what happened mike? What did you hear hi hi mike? We guess we look at some windows issues today too. Oh, what did we do? What did we do?

A

um Maybe it's because it's z, shell right.

A

There we go okay, so um is that c already shortening my shoe on first get is fun. Oh you. You see it all the time, huh scott, okay, okay, so it yeah you're right, it's it's consistent, and so so does it happen all the time. I guess scott. I don't know like so, let's find out so so it happened our first time.

A

Right so so does this happen every time right. So, let's see if it happens, every time why why do you hit this all the time like? What are you doing where you're check oh yeah, okay,.

A

Actually, I didn't get it this second time.

A

I didn't get it the third time.

A

I didn't get it the fourth time.

A

Why am I not getting this anymore.

A

What's going on scott, why what.

A

We hit it the first time, but we literally never hit it again.

A

Okay, so let's just see so while true do dot, slash crd race, sleep, one done: okay, let's print a date out, so we can get some okay. So let's see how long this goes until we see this issue come up again right.

A

It's but we're deleting it.

A

Like what, if I'm deleting the crd, why can't I I mean I can't oh for any crd, the first time you use the crd, but we're literally deleting the cr. Maybe did I not delete the crd.

A

I mean this is a totally new crd, it's it's yeah, so so how long does it take so? In other words, I have to wait between tests then. So I have to do this. I have to do this. Do that and then I have to like sleep, let's see if we, let's see how what the cash clearance is right, what cat? How oh gosh this is so hard.

A

Okay, let's wait 10 seconds and see if it's cleared from the cache.

A

Did we clear the I don't know how long the api server cache caches last um all right. What did I do here? We go.

A

Okay, so again it seems to have passed the first time now, let's see what happens, we're waiting, 10 seconds right.

A

Okay 10 seconds, it seems like it still in the cash, so the cash is pretty.

A

What's mike saying change the metadata change, the metadata name may have it yeah. We could do that, but first I want to see like how long does this cache stick around right? Like I mean, does this cache just sit around forever like what, if I wait like 60 seconds here,.

A

And and then and then maybe in order to reproduce this, we have to look in there so um yeah, I like your okay, do it in a loop and every time make the crd name and short name, have an integer in its name: okay,.

A

All right, okay, maybe maybe all right, maybe I'm stupid, okay. So, oh, oh, oh wait! It said. Look at this! Look at this cube, http cat. What is this? I didn't even know that existed. Wait. What what's this I mean. What is this? No, I didn't nobody told me about this. What the hell is this thing?

A

What is nobody told me about this? I didn't know if I deleted my cache and reran and I could not reduce it is only the first time the short name was ever used that this occurs. I only tried it from one client machine, so I'm not sure this is cluster, specific or client specific.

A

I y. This is weird. If I used a different short name like photo and re-ran it, it reproduces the problem. You can also run coupe ctl api resources. It has the same effect and it has oh so really so if I run so, if I do ls so first of all, I didn't know this, there's a thing called the http cache and that cache what what is this? Okay? So there's a thing in here and it's caching all these requests. I guess it's making some kind of a hash cluster specifically doing a loop every time.

A

Oh, let me turn this. Let me turn it you oh here it is so there's a cache and I don't know like what is in this cache. Is it let's see? What's in here?

A

I really didn't know that this was gonna happen. I didn't know that there was, I didn't know this thing existed. I didn't know there was a client-side folder in my home directory that had a cache in it did you know about this mike? Is that scott?

A

Did you know that every ape every query that you make is cached in a home, dot cube, http, dat, dot, dash cache, and it has the exact query. You just ran.

A

I didn't know that okay.

A

um Yeah, that's a different cache you're right. It's not the cache that you're talking about it's, not the this caching, the api server, it's caching, the crd resource, but he's saying that if he deletes this and then re-runs it he could not reproduce it. It's only the first time the short name has ever ran. I guess what he's saying. Maybe what he's saying is here is that it's not yeah. It's not happening on the client side. Oh wait!

A

If I use a different short name, reran, of course you reproduce it. So we could do this. So let's do scott. Let's do scott's and scott's idea. So, okay, if we say here if we say make a new short name, singular foo, foo, foo,.

A

Okay, so if I do this, will I see it so now I'm yeah.

A

Oh well, that's different because I did foo I'm supposed to do. Foo too foo too.

A

But now I'm gonna do food three, because food 2's already in there, okay.

A

All right somebody's going to tell me I should do a variable substitution here, right, somebody's watching me right now and they're critiquing, my my code and they're wondering whether or not I've ever written a computer program. Before, as I do this okay here we go.

A

Foose.Stable.Com.

A

Delete custom resource this one foo: three: yes, okay, food, three, food, three food, three; so.

A

Delete custom resource and the name is food. Three yeah food, three group ct I'll get food okay, so let me make sure there's nothing in here. Get crd.

A

ah Delete crd foo 3s ctl delete s. Let's delete these.

A

Group ctl, so I'm going to delete these group ctl get crd.

A

Okay, now, hopefully I see it after all of that nonsense, oh well, it's not cleared from the cache. I guess. Okay, I'm gonna make a variable here, because we've done this experiment way too many times now: okay, uh name equals x. Okay, so let's say here.

A

A

A

Name, I'm kind of wondering what scott does where he is checking the crd resource right after he creates it. um Okay, food.

A

A

Okay, phil.

A

Okay, here we go.

A

All right so now, let's try it.

A

Three times in a row it failed, but that could be for a different reason. Let's make sure I use the right short name.

A

Explains read the response from select? Oh, let's see what did he say? It's a server-side cache. Yes,.

A

Yeah, let's take a look.

A

Feel it who is solid? Okay, all right, let's see the root of the problem is that the shortcut expander is run before the discovery.

A

um Oh so, in this particular case, the line shortcut.go and e dot rest mapper continues to rest mapper because shortcut expander read cached data and never reached out to the server, whereas discovery mapper.

A

In client go we'll get a not found error and we'll proceed in validating the cache and refitting discovery in the subsequent call. The shortcut expander is already aware of the new shortcut, so it works just fine.

A

We need to teach the shortcut expander to re-fetch discovery data, but I'm not sure that's desirable, or we could at least warn a user about this situation.

A

Through the problem is shortcut, so shortcut, expander is in client, go.

A

Continues to rest mapper.

A

I don't know why this opened in an incognito meetup window.

A

Expand resource shortcut.

A

I don't know enough about client code to reason about this. You can all you can do. You can do get days after creating and you'll get the same issue.

A

A

So it's only happens. The first time you use the short name. I didn't know it was just a timing issue.

A

So that means this always happens.

A

um What's the short name on this.

A

A

Oh, it's still fo2 because I didn't replace that.

A

What was I thinking, I mean.

A

A

Okay, let me delete these.

A

Okay, kube ctl delete crd that coupe ctl delete crd this one coupe ctl2.

A

A

A

A

A

I should grab for foo right.

A

Oh, we deleted it, that's why so it's not in there and if I go to the crd race and I make a new, let's make a new resource. Now, let's call it y, let's call it yyyy, okay and let's try to run this test again.

A

No resources, so this is a new. This is a new one and I'm not seeing it scott.

A

um It definitely I don't. Why is this not happening on this new resource? Totally new resource didn't exist before this happens, a lot in cross plain and gatekeeper.

A

We may have to do mike zappa's idea, where we do this in, like a for loop or something right where we say vim, crd race and we say name and we don't declare name crossbar cross plane, what cross plane, pla cross, plane, cross, plane, ricardo cross plane. So for folks that join, I see there's like new people that seem to have joined so we're trying to reproduce this issue. 65517.

A

Where and we kind of reproduced it the first time, but I don't know I screwed up- and I couldn't reproduce it again, but if you run this little snippet here on issue 65517 in upstream k8, uh where is it come on where'd it go yeah this. If you run this, you can reproduce this issue at least on kubernetes 19, and I saw it on a two. I saw it on a 223 cluster yesterday. So this isn't something that's uncommon.

A

um Where you create a crd and then you can't you, you can't see it right after you create it. You have to like wait a little while before you can see it, but um one thing that I found out: that was that there is a um there's, a there's, a.

A

Connection lost hold on what happened there. We go, I'm back, okay, so.

A

I wonder if it's dollar sign name I'll, try one more thing.

A

Yeah, I'm gonna try this I'll, try one more thing: I'm not done yet.

A

A

Yeah I in 1.10 echo dollar sign I done. Can I do that?

A

A

What is it this, I'm kind of like yeah there we go. So if I do one of these.

A

A

Name equals dollar sign. I.

A

Will that work.

A

A

Yeah this is mike's test, so vim right. So if I just keep running this, will it ever break? I don't know we'll see.

A

This seems like a really hard bug to reproduce or I'm just screwing it up all right here we go all right. Let's see.

A

I mean I'm making a million of these. My oh I'm doing your thing that you suggested mike. Isn't this what you asked for like like? I, I can't reproduce this again. I got to see it once and I couldn't do it again. Maybe I'm doing something stupid, I don't know so um uh yeah anyways, but I I have seen this before, and um so that's that now um here I'll put my if folks want to try to make this something that reproduces every time. I'll put my code up here. Okay, I'm gonna put my code.

A

If you think you can make a better reproducer than me, I'm gonna put this in the live notes. Okay, this is a challenge.

A

It's the first andrea, live coding challenge ever okay, the first andrea, live coding challenge ever make this reproduce.

A

This issue: okay, so I'll put this in the here's, the hack, md, okay. This is your lead code for the week, scott okay, so here we go what happened? What's going on, many k8s will be going down with this one cve and next cv will be making windows jokes shout out to daniel magnum, indeed cross plane. Folks, did nice work?

A

Oh oh, I see because scott made an ingress joke scott's, making fun of rick, okay scott's, making fun of ricardo. Okay, I go all right so if folks want to try this, I'm going to put this in the show notes. So I'm going to put this file, I'm going to put it in here. Okay, I'm going to put it right here.

A

I just made the readme okay, so there's no reason. This is super easy to remember. Where the show notes are you just go to ja unit, 100, k8s, prototypes, andrea, live 427 right and see if you can make that right, reproduce it and the bash snippet that I was using. Is this one for those of you that don't know bash?

A

ah This is the snippet okay. This is the test. Snippet, okay, right and then see if you can make that cr make that crd race script and then see if you can get that to get that to make that thing. Do that thing? So everybody knows okay, so, and anybody who can get this to work, we're going to send you another android t-shirt, another one, because I'm sure you all have t-shirts by now. So um all right!

A

Where were we so the other thing? What was it? It's almost. We only have 20 minutes left, so I don't know how we're going to crash that cd with the damon set in 20 minutes. But let's see so so I mean the first thing I'm going to do is. Can I can I google for a solution here? So if I do like damon set with.

A

Curl, like is there something that will is there a daemon set that I can with api server post.

A

I mean I guess I could do a damon set with a bash coupe ctl script in it right.

A

Let me see here.

A

Let me see if I can find cube ctl in a pod right. So let's see.

A

Yeah run goop ctl, here's here's one I can borrow- I don't know what yeah here I found one um it next dot io here we go. Here's one I can borrow right so looks like looks like this. uh We've got a this is 20 20, I'm assuming this might work. So, let's see if this works, can I do this. So if I do them that thing's probably on I don't know if I'll get talk rate limited or not, um let me okay, get fetch dash.

A

All get status, get add dash, you get commit m, go update for multi-node, okay and then I'm going to do git pull dash dash, rebase, okay, I'm going to pull down the latest and then I'm going to cd entry, light cd.

A

Cd, andrea, live okay, so now I'm gonna see the fcd thing that we're talking about so cd4 27. I'm gonna make this bot v, I'm.

A

If anybody else has a snippet of how I can make a pod. In a shell script, coupe ctl create dash f, t max new dashes t mux, a dash t.

A

Okay, so cd dot, dot cd source case prototypes.

A

A name for your hackathon: you can now you've got it now. You've got now. You've got a project yeah, well you and ricardo, um and then after this I'm going to do I I know mike's here and we were talking about windows stuff. So we figured we'd look at some stuff, um we'll look at the end of the episode at what is going on in upstream windows. Maybe look at what issues are open. So uh someone in oh, let's see what did ricardo said? Scott, you don't have a t-shirt.

A

Well, I'm gonna make a deal with you, scott, we're gonna. Send you two t-shirts.

A

If you can solve the coding challenge and we're gonna, send you one t-shirt if you can't because you're one of our favorites, so um we're gonna have to tell we're gonna have to talk to susan and we're gonna have to say susan susa, we're gonna have to say susan. I need you to tell me why scott does not have a t-shirt, get pods dash a did we okay, so we got it. We got that internal coupe, ctl pod, so coupe ctl.

A

uh What do I do? Coup ctl exec dash t dash, I internal cube ctl say: do I get a shell in here?

A

Oh, what is this unable to upgrade connection?

A

What is the name.

A

What white? Oh it's not! Oh, it looks like it's restarting, it's not happy.

A

That's not happy, oh it didn't it's not happy.

A

How can I get coop ctl in a pod without using this, without using this thing that I found on the internet?

A

Anyone got an idea.

A

uh Api servers.

A

Is there a uh let's see here.

A

I see that there's somebody once made this. um Oh no, that's that's stress, testing api server. um Anybody ever use kaboom.

A

Has anybody ever used this before.

A

Does it push at cd? Is the question uh running skill launching to see these aren't real? These aren't testing etcd. This is a thing. I don't think it is. I think it's just seeing if it can, like I mean you know like I think it's just seeing whether it can create and delete I'm just going to do. Coupe ctl patch, a million times coupe ctl. Here we go coupe ctl patch example: let's just grab a snippet okay, let's just grab a coupe ctl patch command right, let's grab one of these here we go.

A

um They've got one right here.

A

Yeah can use this, so if I get this deployment right, if I make this deployment, I like this example, vim patch vim, deployment.yaml, okay uh-oh, I screwed up set paste here. We go okay, um coop ctl create dash fd.yaml, let's see if that comes up now.

A

What I'm going to try to do here is I'm going to see if I'm gonna see if this comes up and if so, I'm gonna see if I can just continuously patch it and um as long as that, but I don't know how this makes me start thinking about caches, because I really wander now with this caching thing like. How do you even know?

A

How do you know if you're really pushing ncd unless you're, actually looking at the brights per second or anything, you don't really know so. Their idea here is to next to a json, merge patch on your same deployment.

A

Here we go okay, so now I can make a patch.

A

Okay, so I want to see whether I can.

A

A

See if these two work in isolation, right, ls.

A

Coop ctl: what is it patch patch deployment patch demo and uh dash dash patch file equals.

A

What did that change group ctl, cube, ctl patch dash dash help uh grab file?

A

Oh it's now file name.

A

A

A

Coupe ctl patch.

A

Oh, maybe it's that.

A

Cannot unmersel string.

A

I don't think that works. Maybe they don't want me to send a file.

A

Maybe they want me to send dash dash patch dash help.

A

Oh, let's see coop ctl patch deployment dash dash help.

A

uh Patch to be applied.

A

Maybe I need to just pipe that file, so can I just do patch.yaml.

A

Into this, can I do that and then.

A

The patch to be applied, I don't think that. Well, maybe I need to just make this a piece of json. Why don't I just make this json, since it wants json, so bad, fine I'll just make it json yaml to json, and now, let's see how many ads this puts on my here, we go.

A

Okay, here we go all right, cool, so patch deployment and then coupe ctl get deployment.

A

Patch demo dash dash patch right and I'm going to do this, and can I do single quotes in json so that this works? I assume I can alright. Okay.

A

So this would be, I suppose, a potential start to a daemon set if this works.

A

That we can just run like a bash script, thing that will continuously forever and ever continuously randomly patch um a deployment, but I suppose you'd want to make a new deployment for every every container or whatever. I guess. You'd probably just want to do a single go program that spawned a bunch of threads. But then uh I don't know, then, if you were only you'd, probably want to run into damon sent since you could distribute the amount of looking for an object of type key string.

A

I wonder why it doesn't like that.

A

uh Maybe I have like some hidden characters in here that I didn't see. I don't know um okay, so I'm I'm gonna, I'm gonna. I'm gonna end this in a second, then we're gonna see if we can look at I'm gonna, maybe I'll show we can look at windows.

A

We can look at.

A

Mike was interested in some stuff on the window side. Looking for the beginning of the object, key string invalider, I don't know where that slash. Slash thing is coming from like what, if I just did this, what, if I just did a patch of nothing, would that work?

A

What if I do this.

A

Okay, so that worked, so why is it? Why does it hate me? So maybe the reason it hates me is because I can't do these single quotes. That's the only thing I can think of so maybe if I do this it'll work.

A

I don't know we'll find out.

A

That looks happier now than it was before. I don't know, let's see, oh no, that was the eo left. That's not any better!.

A

Okay, let's see, let me ch: oh, it worked okay cool! So all right, we have a little tiny, coupe ctl patch command, at least that we can at least use as a little template to see if we could get this did I. Where did I put it? I put it in here right so now. Let me put this in here right, okay and now the second ant live coding challenge ever right.

A

Can we can somebody take this stupid patch snippet put this into a pod with a coupe ctl, wrap it in bash and see if you can use it to continuously patch a bunch of deployment objects like randomly create what like say a hundred deployments.

A

Put this patch into cube.

A

A

Let's patch those those deployment objects and then as you're doing that um monitor to see if, um before every patch have every node re have every.

A

Every patch have the command cache the hash of.

A

Oh, I don't know if, oh I don't know how you do that, so I don't know how you're detecting consistency right so you'd have to think of a way to detect an inconsistent like you'd have to this yeah. It would be a damon set and eventually you would detect, I think, an inconsistency um and see if eventually.

A

One of the nodes to detect an inconsistency you'd have to get to a point where one of the nodes, um one of the nodes, would.

A

Would at some point yeah it's like. I don't know whether you would need to synchronize all the nodes like you might actually need to be smart about it. You might need to like sort of have a you might need to have some kind of a scratch board where you wrote down.

A

A

This, the current value um of an annotation and sort of continue printing that out, I come to think of it like see see if you can see if you figure out a way to compare the object, as seen in the script over time, depending on what node in the game and set was running a get on that resource. That is constantly being mutated, see if things get out of sync, so I don't know it's my first attempt at trying to think reason about how you would reproduce this um anyways.

A

So that's that um jun, jen, hey so jin jin. We we we did uh you get three someone on entry, a t-shirt, just fine new, um I'm sure ji and jen will be giving out kubecon t-shirts at kubecon, andrea t-shirts, api server in case you folks, don't know: android leverages the k8's highest pierce build yeah yeah, it does so. um Yes, it does so so andrea.

A

Does that and then guess what you know what else june jen one time we were talking to the carvel folks and it turns out that cap also does the same thing. Cap also leverages borrowed from andrea and all does the same thing to sort of leverage the leverage, the api server in the same way and extend it um cool. So this is just like my first step of trying to think of like okay. Here's all this is a patch command, but I'm like okay is there a way we can? We could script this.

A

We want to put like a variable in here kind of the way we did before right and we'd want to just keep patching it over and over again until maybe we were able to see an inconsistency but there's some kind of thing where we'd have to record the state, or we would have to do the thing that I showed you all how to do. Last time that we played around with where we were running at cdctl um and uh what was that trick? It was um oh, what was it um at cdctl right, yeah?

A

It was uh endpoint status, this one right, so you might want to do you might be able to do something like this right and you could you could see whether things you know check whether things get out of sync with that right, I don't know. I think I, like bro, wrote a blog post about like with the snippet on there. um No, I no. I didn't write that up.

A

Sorry, yeah, okay cool, so um what we were trying to do was reproduce some lcd consistency issues and look at how we could do that and we were able to do it.

A

The first time we were able to see that that's not an fcd issue that that we were able to see the first time that we were able to reproduce this crd related issue, where the first time you create a crd, the short name sometimes is not available for like a split second in time um we were able to reproduce that once, but we were never able to reproduce it again, which is weird, um but so yeah good.

A

First, two andrea life coding challenges ever so uh so uh mike zappa joined also- and we were talking about. What's going on in the windows world, so issues windows like let's take a look and see if we can do some triage, really quickly: cool github.github.com.

A

Kubernetes, if I go here and I go here.

A

Let's see if there's anything interesting, that we can look at that's tagged, the sig windows, sig windows and- and maybe we can look at some sig network issues also right, github.

A

Kubernetes um and let's see what we've got for sig network issues um and let's look for ones that have assigned to nobody right. Let's look at these because these are the ones so folks want to get involved right. Here's how you do it right! You look for things that aren't assigned right, um so some of these might be tagged as like good first issues, um but if we look briefly, we've got 60 open windows, ones and 112 sig network ones. So I don't know like let's look at this one.

A

Let's look at oh god contract. So it looks like we've got. A lot of we've got a lot of issues.

A

I I thought that we had sort of got to the point where, where um let's go here, sig network yeah? I think this is what we want to do. No s e sig network right. So I'm surprised that there's so many of these so now label. So then you go label sig windows.

A

Here we go sig windows, so let's do that and then let's go to this one, let's get rid of that. No, I assigned e, so ed. Okay, these are issues that are not assigned I'll put these in the well. You don't need these show links. You can figure this out. You all know how to use github.

A

So I see 45, open sig network issues and 34 open sig windows issues. So if I'm going to look at these cannot modify registry keys during windows, pause image, build process, that's mark.

A

That must be as something in the build that got broke. I want to see if there's anything, networking related soap tests for azure k8 clusters with windows, nodes, marius disks become read only on windows, node, divya and patel. This is a interesting I think I think divion saw this when testing vsphere csi when you rebooted it. So that's an interesting one. If folks are using vsphere csi proof network policy test reliability. So is this a a meme? You opened this one. What is this? What did you fix this.

A

This is a good one I think mark. uh This would be an interesting one to start. This is a good getting started issue. I think um so.

A

Decreasing the tcp re-transmission time out of agn host. So in order to do this, you would you'd go into the upstream network policy. Testing suite well you'd actually go into the agn host container right, so the agn host container lives.

A

No kubernetes kubernetes right the agn host container lives.

A

Where does it live? Damn it agn host lives.

A

No, that's not it is it in test images, maybe test images here it is so the agn host uh container is. This is an interesting one mike. So if you go in here, you could play around with the agnost container and you could see whether or not.

A

You could go into the yeah, you could go in and you could see what, in the network policy testing suite. So I'll show you that one. So that's.

A

I think I've got it yeah I'll just go in here, so cd source, kubernetes.

A

Oh god, it's so slow all right, we're a little over time, so I'm gonna. But this is a good starting point I think mike and then you and me can sync up later on afterwards. But if you go into test e2e.

A

A

A

And you go to networkpolicy.go right. If you just grab through here, agnhost dot, slash, you can see what we do in the network policy and end test. So me and amy could show you this.

A

If you wanted to learn any any of us could show you this that work on this stuff um so and uh so these print out these tables these connectivity tables- and you could go in here and you could look at how this works, and this goes off and it probes you have agn hosts on different pods and they probe each other, and so you could go in here and you could sort of look at this windows issue where it looks like we have some flaky network policy tests.

A

So we wrote, we updated these tests to support windows recently, so you could spin these up on a windows note and they'll probe each other, but we don't do udp, because that somehow crashes, everything um and I think there's another is, and it would be interesting to try to reproduce this and see if you can get uh see if you can solve the reliability issues or reproduce the reliability issues that folks are seeing when they rerun these. So to run these tests, you would compile kubernetes and then you would you do make you do make.

A

E to e dot test right, so you'd run it like this right and then you do like make what yeah this? What right? So you do that. So that's how you would those are the two commands you would need to like to sort of reproduce. This I'll put this in the show notes too.

A

Where is it? Andrea live no where'd, I put it where'd, I put it. Entry live get uh yeah here it is so where is it here? It is so I'll put this here right. So now you could go in here and I'm gonna put the snippet of you know. This is roughly what you would do to run. Those tests and you'd need a windows cluster, and for that you can use rc windows dev tools, and then you can yeah, you can go and you can.

A

This could be a good starting good, getting starting issue getting started on advanced windows, e net working policy, test improvements, and these are fun tests to run because you learn a lot about kubernetes network. Let's see what running these so, um okay, cool, so yeah, that's the that's! Today's show thanks folks for coming. I see four people hung out. um Oh wait, I mean said we're fixing the timeout close.

A

I need to drop off yet thanks vivek. Thank thanks. uh Everyone yeah for sure cool, I'm going to drop to david thanks man thanks thanks, uh yeah everybody thanks for coming.

A

So, uh let's catch up next time and who knows next time, I do this, maybe I'll we'll try the ncd thing again, but if folks are interested in trying that little coding challenge like I was only half joking about it, I think it would be a fun thing to try to build a thing that detected those fcd inconsistencies and and spammed the uh api server through all the way through to fcd continuously in the damon set.

A

um The damon sent might have to make go functions, though, because you may need to have like multiple threads running. I don't know. Okay, see, y'all uh enter your life thanks for coming. If you all are having fun when you uh come to the show and you're learning stuff make sure to like and subscribe so that we can, uh if you enjoyed this, show like and subscribe, so we can justify doing it more okay, cool right! Thank you. Everybody see you next week with chinchi.