Kubernetes SIG Windows, 16 Oct 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Windows 20181016

Description

Kubernetes SIG Windows 20181016

A

B

A

Everybody and welcome to another sick windows, Meetup I, think you all for attending I'll turn the mic to to Patrick and and get started. Thank you.

C

Okay, so I've got a couple things I put on the agenda, so one that Michael just asked about here. What's the latest on that just testing instability I still don't have a public date yet on when the Windows Server 2000 19 media will be released.

C

I think it may have been available briefly, but then it was pulled due to the issue that was found on Windows 10, and so they, since they share a lot of the same files, they decided to pull them both while they were I'm addressing that issues, but I think it should be.

C

You know, in a matter of the next week or two before that's ready again and then the other thing that I've been doing related to that is I've basically been trying to collect all the different tasks that were needed in order to move to Server 2019 and a nerdy cut that a project board listed there, then we've got cubic on coming up. um Anybody have any other, because you want to add for today.

D

Chloe, you can two brief topics about some. Some testing patches we've been sending back. Okay,.

C

All right well I had testing. That's the first thing on the list. Do you want to start with that? Then.

D

Yeah so cloudy, if you want to take over and talk about your patches.

D

Yeah, of course, I can't beam him separate anything. His.

E

Just ii saw a couple of matches that.

E

One regarding yemen, fires, output images in DMM files, which should get fixed another one regarding some tests, setting spawning odds with the host network enabled when they don't even meet them, which basically is a blocking issue on windows, because you cannot spawn containers from windows using host network.

E

Thirdly, there is regarding strongly no specific commands which should be made more agnostic and there's a pull request for that, and the latest one is regarding being able to skip windows. My related tests in code, basically ginko, can accept.

E

Argument not os3 strong and it's actually used for a couple of tests as well like if it's debian or if it's anything else, it's gonna skip some tests. We can use the same mechanism for Windows as well and a couple of scenarios in which some tests are trying to to spawn some containers, which are incompatible with how it is at the moment like spawning with the SC Linux options or trying to run with you idea or a JD on Windows or transform some specific Linux containers, images on windows so on and so forth.

D

Second, so, basically these these tests, the college, is talking about we're skipping, we're passing the regex to jingo to skip them.

D

But this way we can just say confirm, run conformance no performance and it was given all automatically so there's no, no huge skip test regex. We we need so far and if I'm not mistaken with this patch I think we can also but correct me if I'm wrong value, I'm referring to same thing, we can also have conditions inside the test.

F

D

If you want a specific command, if it's on Windows.

E

Otherwise, just skipping.

G

E

We can easily check that.

G

E

Also setting a precedent, so future developers will take it. Okay, if they like this, if it's, it should be passing on Windows or not mm-hm,.

C

So, just something that we would need to set.

C

D

C

So it doesn't automatically look at the node. Oh.

C

D

Because keep in mind the tests are running from a separate machine, then the container they never touch. The cluster itself.

C

Was it could query the nodes that were in the cluster and then make a decision.

B

C

Is gonna be a diffusion test cases fort knows what nodes are there.

E

This was made due to do other presidents that I mentioned in the beginning, since there were, there were already this, which was looking in looking at that parameter if it was Debian or other this units distro, it's basically the same idea and it's we would be easier to prove in Excel. Ok,.

C

All right- that's excellent. I was just talking to two lucky I'm, even sin I'm about this as well and and he's going to help help help her out this discussion around skipping, the conformance tests that don't work on Windows, and so it's I'll add him. Did this PR as well, so.

B

C

Gonna be doing is failing an issue that basically lists about all the tests that we've already looked at, which is the red Jack skip list that we had before then work with him to get that on on the right agenda and then we'll be able to reference this, as as, as the proposed way to fix it.

C

And does this cover all the ones that are currently in the skip file, that's over on the ete repo, or is this just some of them?

C

D

D

Think it Skype.

E

Should cover almost all of the conformance once okay I haven't taken a look at the full run, because there will be a lot more in that on the site, but this conformity should be almost all down. I'll double check to see if any other new tests were added, which cannot for Windows should be most of it.

D

The only downside, to put it so with this approach is obviously, if not you unscripted fix it or whatever. You will need to go through this whole process of reviewing approving, but at least for the tests and changes minor by minor changes like this I would assume they would be a little bit faster.

E

If you want to check if something is fixed, so you can just run the tester so different that I know that was destroy. That.

F

E

C

Did you have any chest update, said I know.

D

Her so we're now looking at that fix, you are talking about at the beginning.

D

For the stability issues we we were having with triangle which to properly apply that that states, because darker basically doesn't want to start after it with trying to see how we can let's say for starting or figure out the issue for that one. Basically, after a reboot doctor, won't start and there's a SS engine issue. For that and correct me if I'm wrong, I think you referenced another issue into that issues. So probably.

D

This doc, not starting after after reboot, maybe maybe.

D

A more a bigger problem than we know, but honestly, two to reboot the nodes. We don't really need it until now that we have to apply that fix other than that, there's no need to ever a good, a nose for festivities. Okay,.

A

So we so when doctor doesn't, when you say doctor doesn't start, is potentially a problem. We something more windows or docker itself, like I'm, just wondering if we have engaged with the docker team to see if they can help troubleshoot that at all.

C

Well, I think that there's I think that we may be able to mitigate this by just setting the right service, restart action and setting the right service dependencies. So I think it may just be an ordering problem or.

A

It might be an ordering problem, like I mean what do the logs say to the logs? Have anything I mean usually when I say restart.

B

C

It's an exception in.

A

The logs or even log, or something like that, yeah.

C

The logs said that the service wasn't ready after 30 seconds, and we talked to the National Knight earlier today and he's thing said since the patch is still installing right after Windows comes up. If that, if the Windows networking services and ready, then docker may hang for some time waiting on that.

A

But that's usually only once right, the patch will only be installed once if you go through the same scenario, same stress test and you try the game. Then the patch ideally should be. There does a service start in that case,.

C

I haven't tried after a second reboot yeah, okay,.

A

C

The service it works, okay,.

A

So do monitors okay, so that definitely is potentially a timing issue.

A

Maybe you can who can I guess on doctor installation that doctor decides how long it should wait to start and their service configured they're doing that control right. So.

C

That the 30-second timeout comes from a service Control Manager.

A

Install the doctor installs on Windows, they define how their service will start. We'll start the ability options they have and how long today.

C

Oh yeah, we could, we could set the late start to sometime.

A

Another option is: maybe our services like for the cubelet can depend on doctor who, just if I remember correctly, does depend and then our service can have a delay start and then it will force doctor to start on its own like two minutes later, for example. So that's an option you can do. We don't have to notify the doctor service. It just works.

A

C

And right now the cubelet doesn't have that dependency. So that's definitely something that we should try and that's that's one of the fixes I've got I posted a link to that. It's you I filed, so we'll try that out. Okay.

A

Son sounds good.

B

No sorry I just joined okay.

C

No problem, okay, so I already mentioned a server 20:19 rerelease I'm, still waiting on on that to show up and then ice looks like you, jus, just added something as well about test results. Did you walk over that now, while we're on the testing topic.

H

Sure yeah, so we've been trying to make some progress on the testing and jce and Peters doing the the majority of work. But I created a cluster manually using the Peters guide and I ran the III test against it. So I thought I would share with everyone about on the dock and see like the set of tests that actually passed versus failed, so I didn't skip any tests and I ran conformance and no components tests. So just as a reference point, if anyone is interested in the King, the results are in the doc.

C

Okay, excellent yeah, so probably the that, let me find the exclusion list for you real quick, because that would be the easiest way to to to see if the failures are known ones or not.

D

So, at the quick glance, while Patrick is pulling the exclusion list- yes, some some of the ones here in the failed tests are excluded tests. Oh.

H

That's great I'll, look at the list and figure out if there's any difference in anything that should have white and not mm-hmm. So.

B

Then I look at the network test, at least I think the DNS ones we are. We understand the issue and we are going to go to Signet work with the understanding how why Linux has done that way. But there is a patch that Adaline, like the Adalind, did where the last four tests related to intro to pod communication. Note the pod communication. They were requiring the host network in the test.

I

B

Did a patch where she removed it, so you should be able to run those four tests to talk with the dot patch and they should be able to pass yeah.

H

I see I just used, yeah I just use whatever is the test right now, so I can just patch it and run it again. Yes,.

I

H

D

We are using perchance, patrick, has a guide for testing and so on and so forth. Are you perchance using that I.

H

Use sort of a combination of your repo in Patrick's, so I did have to patch a little bit and then just run it against our local cluster and using the repo list that you provided mm-hmm. So for the test.

D

I think some of the best patches makes a both from today are also into the kubernetes from our repo, and it will win end-to-end win. So if you build the test from that one, you probably have all the all the patches and will will also pay with the the patches we we send today, I.

H

See so most of the patches will eventually be upstream today, so I'm, just using a.

C

H

C

Yeah, most of those are open, PRS right now, mm-hmm.

D

C

So I posted a list of those from from Claudio into the meeting notes, so you can pull them back up later. Hey.

B

One curious question: did you see any notes going up and down or you didn't see them I.

H

Didn't see those but yeah I had to disable the auto update. If that's me, yeah.

B

Fair enough, okay,.

D

Are you able to give us provide some logs for this because I'm curious? If we see anything in the cubelet.

D

For the tester should have passed.

H

I have huge cubed, logs and I. Didn't I didn't do a good job of like collecting those automatically, but I should be able to get it. Yeah yeah.

D

Yeah, okay, um yes, there are um well. We have the dogs and there's no point in talking about it right now with, for example, there are some tests that requires some folders to be on the host, temp, folders and stuff, like that. So that will probably give you some extra to tests. I see.

H

Yeah, could you leave a comment on the dock as well, just so that now yeah.

D

Okay, awesome thanks. Thank you.

C

Okay, excellent yeah, thanks for sharing that and if you have any other questions on that, you know be sure to to ping us around the Sigma additional and be great to compare the results there mm-hmm and then this is using a GC, specific plugin. That Peter was working on for the networking right now.

H

This is actually just using the the win CNI plugin that I think you guys also use, and so we just have to set the pass ID erratically. We still cannot get it from the middle object and then we clean right now. Peter is working on that. Okay,.

C

And so this was Windows Server version 1803.

C

B

C

It's a static yeah.

B

It's really good cool.

C

Excellent okay.

C

So move on to the next one.

I

Hey I just bring up something quick, so I posted a link. You get a pull request around an issue we fixed in Dhaka and in Iran he starts. Basically, if there was a transparent network that happened to be there and before she start was issued. It.

E

I

E

I

Indian to basically just hang because it kept looking for an empty field, so that was the issue that was recently fixed.

I

So not sure if this is what this could be, one of the things that's affecting what we start.

F

C

All right, thanks!

C

Okay, let me he sends a note. So I don't lose it.

C

Okay, so the next thing I had on the list was I was going back through.

C

Through the work that I was doing to bring up Windows, Server, 2000, 19 and container D on India's, your environment and I, basically put all that into one project board I'm using to get work signed out to multiple people, and so the stuff related to you know. Acs engine is basically work items that are specific to how we're deploying to things on donnager, but all the other compliance performance blockers are in there as well. So I've got a list of everything.

C

That's waiting waiting to be merged, Claudio's issues are already listed there and then, under the conformance blockers, I've got a list of the other ones that are known there and then the stability bucket is where I'm actually gonna be refocusing to spend, spend most of my time so I'm in the process of getting these other issues assigned to some other people at Microsoft, so I can focus on stability testing.

C

Yes, if anyone else would like to you know, you know work with me and start adding more. You know windows feature specific tests like I'm, not looking for things that test Windows differently from Linux, but what I want to do are add more tests that are, you know, targeted to be run on Windows, so that we can start looking for some of the potential stability problems that that we've been that we've been hitting.

C

You know that James has been adding a few of those over on the ACS Engine side as well, but I think we should probably start getting some of those into the kubernetes repo and then in the work under engineering blockers are over there. That's, basically the things that are needed for us to get this little test pass up running with a nerdy, and so some of those are things like you know. Getting the container D builds fixed daily.

C

That's something that the the container demon painters are working with some engineers at Microsoft on and then we'll need to integrate those into the setup steps and then same thing for.

C

For the kubernetes branch as well, so what I think we're going to do is just like how Italy has been collecting all of the test fixes in that branch that I have listed there. We're probably going to do the same thing for met ease where we have something: that's basically taking all the PRS that we're waiting on to merge, put them into the same branch and then build it. So we can at least measure whether each successive PR is continuing to fix tests right and not cause cause other problems.

C

So I'll be working with with someone else on my team to get that that branch set up. So we can get that available as we're bringing up in Windows, Server, 2000, 19s and then the stuff under project tasks are basically the things around finishing to bring up the the.

C

Conformance test adjustments that are needed to exclude the stuff around windows and, as some of there were these type tasks that are done there.

C

So, if there's anything that that any would like would like to help on or if you want need any other items of sign, you know please, please just know, but right now. My biggest concern is that when it comes to stability bugs since we don't have any test cases that can hit these you're consistently every time, it's really hard for us to know whether or not we've actually fixed those problems and then in like YouTube when you're testing over on GCE.

C

If you start getting any issues, it would be good to compare against this issue if already known or not. Otherwise, we all wish could induce things. I'll do an edit repo.

C

So that's what I had for that container D stuff, some pushing to have those most of that stuff done and I think we should be able to get it done in the next one to two weeks ele running with, and then that's going to allow us to actually test the other fixes that we had for things like single file, mapping that are only going to be feasible.

C

C

Anybody have any other questions or anything to add, add in that area.

C

C

Then the other thing I just mentioned is that we we do have the other two cube con events coming up, but the one in China there's going to be two presentations.

C

Ping Fei from Microsoft is going to be doing a talk on Windows containers and then Cindy from Huawei and Dinesh are going to be doing a presentation on Windows networking.

C

Very cool men in Seattle, I'm gonna do a quick one on the developer experience along with one of our developer advocates Jessica Dean, but that one we're gonna try to focus on the higher level stuff like draft and how so it was got any. You know, suggestions on things I'd like to see in those talks forever.

G

We had some fixes that went in recently post those um those are related to networking. Specifically, we saw an issue where load balancing wasn't working correctly and we were always like setting traffic to the same pod. So we had.

B

G

For that one then recently, and as part of that, actually um we also thought the approvers lists updated to such that think for a coup proxy for the window. Specific coup, proxy binary, Ganesh, Madonna and Peng PHA are the approvers and reviewers. Now, hopefully, that will accelerate like any future fixes that we need to approve and get merged by the right people, but that's good to see, and then the other fix that also went in is port mapping.

G

Support for RC, nie I, don't think any conformance test or any additional conformance tests are were blocked by these previously. But it's good to see that we have six is working.

B

So there was another fix in Azure CNI where they were not propagating the DNS suffix right. That is all so that basically helps some of the best to succeed. In conformance, you should probably link that as well. David yeah.

G

B

One that I did through namespaces.

G

Yes, yes, oh yeah, I think it's a beautiful yeah.

B

I think you one should get that successful when she is running the conformance yeah. It's in.

G

One dot false next, our next ACS engine release will have this or you can build it from sources now.

D

For the tests I'm building it since there's no okay.

G

D

C

Okay, excellent just takes photos updates.

G

Fixes this issue that I think Claudio filed yeah.

C

Okay, great alright I got those in the notes all right. Well, thank you. Everyone, I'm gonna head over to this. Did you have anything else? Michael I know nothing else. A.

A

Budget just YUM I'll send you my cellphone. If you can ping me later on, I want to try to you about a couple of things. I got your signal. Yeah write.

C

To me out so, I can actually thank.

A

You all appreciate.