Kubernetes SIG Windows, 28 Jan 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Windows 20200128

Description

Kubernetes SIG Windows 20200128

A

Hello, everybody and welcome to another see windows meetup, it's their 20th of January and, as always, is a recorded meeting. So please adhere to the SNCF rule of contact. It's actually funny when I always say that it's like we don't all become like you know, idiots and start spitting profanities when it's not recorded. But you know, rules are rules. I have to say that. So today is the enhancements freeze date.

A

So today is a day that we need to make sure that all of our enhancements and all the caps Irene is there I guess Patrick already mentioned this. Are there any cab merges left and all the cube ATM when in.

A

Deep, did you get the GM I say I haven't seen any anything on that is the bond.

B

So I did update the issue on there saying that we were playing on promoting that one and run as username to stable. We don't actually have any kept modifications there and so I think we're good to go on those because we're just following same criteria. That was already there. Okay.

A

A

There is a place in then I might be wrong. Don't wear white I.

A

Swear there was a spot that you had to put in the enhancement as well.

A

But I'm I'm probably wrong well.

B

I think the main thing is. We seem to make sure that it's on the feature tracking spreadsheet, that there you're sorry, the enhancement, tracking spreadsheet, pin and a tag. All this stuff has tracked /. Yes, so I'll ping them and slack and see if I can get a gale link to that yeah.

A

They usually pick that up. That's why they ping us on some of the issues and when we tell them where it's gonna be and put the milestone and that's when they do it.

A

C

A

Be go based on the issue, so if all our issues are tagged and I saw you tagged 118 for a couple of them.

B

Yeah, hello, I'm just checking here: okay, so here's the enhancements tracking sheet.

B

So bonnet Irina's group is different. That's Linux! One.

B

Okay, so GMS a is listed as tracked.

B

Container DS listed as tracked run as user name is listed as tracked.

B

Runtime class is still as tracked.

B

Although there's still the PR that I'm discussing with sig no dumb in a little bit that has emerged on that.

A

You know so CRI container DS also tracked.

A

Did you come? Did you commit that for for alpha on 118? Why.

B

I updated the cap, it's still, it's still going to be tight, I'm, not sure whether or not I'll be able to get it in when I'll be a before code. Freeze or not, but I think we're down to a relatively small number of fixes and.

B

Yang and lantau got a test pass up and running, it's not passing consistently, so we just need to basically get past a few bugs to get that.

B

I can plan on giving an update on that next week.

D

A

Got it by the way? What issue number is Jim Irsay I couldn't find that in this pressure right now, which one, what what's the number for Jimmy say.

B

A

8 9, um okay, that's I'll, step in the full name. That's why.

E

A

So it looks like we're in good shape if there's no caps, I mean.

C

A

Really there was only one kept I'd like to worry about.

B

Let's see it's been back yet, no, no.

A

C

It's not for a few more weeks actually yeah, okay,.

B

Maybe you know this one gap so so briefly, I said pinged a few people I said we got Jocelyn on Colin Dinesh I'm about the cube ATM stuff, and so they they were look. They looked at that kept that Ben and Gavin, updated, I. Think kind of the one I've seen in question that we needed to talk about there was that we didn't close on was whether or not we were gonna use wins as is or do something like pulled into a sig repo and and make modifications there.

B

The one of the concerns that Dinesh raised about it is, it does have a lot of specific, like HMS, calls that basically, he doesn't think they need to be there for the use case, we're proposing for running daemon sets and then being able to run a host process, and so I think removing those would be important from a maintainability standpoint and also reduce some of the risks.

B

That's there because I don't think you were going to use those for the CNI stuff specifically because the process was going to run on the host and so I think we kind of need to make some decisions on how on how that goes forward and then and I think. There was also an outstanding question on how we lock it down whether or not you know we need a pod security policy or something like that to control access to that mount, and so yeah we've got a lot of those things kind of detailed in that doc.

B

We just haven't really kind of now that we're getting too moving forward on this reasoning, sort of decide what what the plan of action is for, where that codes going to go and how we update it.

C

In the cab we do call out that will provide a you know, a pot, a pot security policy expect that they can use so I think that one's an easy, easy one. As for we spoke briefly about you know maintaining our own forked when us and we have an action from like January 14th to ping, the winners maintainer to see if they're willing to limit or remove calls mm-hmm I personally haven't done that I, don't know of anyone outside.

B

Yeah I, don't think anybody's been looking at other than you and Ben. Okay.

C

I can I can take on that action. Just talk to Roger folks, I, don't know if we want to discuss maintaining our own separately in this meeting as well like right now,.

F

Do we have any concerns around updating, binary.

F

Process launched through winners are.

C

You talking about the kubernetes binaries.

F

The CNI specific binary that might be launched.

C

We we haven't thought about updating, like the CLI, binaries I, think for the the kubernetes binaries, we're hoping they just follow the instructions for obtaining a linux worker where you have to stop the cubelet I paid the binary yourself and then just restart.

B

C

Cni is something we would have to think about.

B

Yeah I mean it seems, like you need something to make sure that there's no instances running before replacing it or something like that.

B

It's almost like you would need a lifecycle, script or something that's moved copied onto the host and then run there. Oh.

A

You can always use draining right.

B

We're talking about the windows processes are running outside there, though, because when you close the dam and set it doesn't close the host process, that's.

A

C

You I think would win as the container will constantly.

C

Okay send out, I could keep alive and the server relies on that and the moment it stops receiving it'll actually closed the the host process. So, if you were to, in short, if you were to delete the container, it will actually close out the host pasta.

B

How interesting I didn't realize did that? Okay,.

C

And the see nice stuff is going to be shipped out has basically a docker image, so I think we also need to talk about hosting that as well.

C

You know one one brief idea was getting flannel to supported, but then we also realized that, as windows get gets updated every month, we would have to sort of keep that up to date as well.

F

F

Can just call out one security aspect: I'm not sure this is that review or the security aspect of the review has happened already, but I think, like you know, given by the CSI proxy me, ran through some of these very similar security questions. The overall idea of a host process being spun up to a named pipe. Was it it kind of drew some interesting responses where people were a little uneasy with it?

F

So I guess if that, if that movie has not happened already, you know make sure that that aspect is pretty well guarded with PSPs and things like that, so that you can address those as they come up.

A

We did the review for you deep who, from six security, was it Jordan for someone else, Tim.

F

All clear and okay.

A

So good gapping says security channel before maybe you need to ping those guys privately and say hey. We need to talk yeah.

C

I think Jordan. He said he didn't have the capacity and also didn't have much of the context, but I think Tim. All Claire was another name that he mentioned so I'll. Try him deep anything else that came out of like to CSI approxi, like other than the named pipe yeah.

F

Pretty much I think the main concern from storage was I. Think initially we were exploring a very similar mechanism which is like. Can we just expose an exact? You know system like a API which says well execute this PowerShell script for me on the host for me as and like on behalf of the container, and that could be, you know super generic. Doesn't it is not doing anything, storage, specific or CSI specific, so it could potentially be used for any other purpose, including you know, H and s configuration size.

F

You I think we definitely did not want to like. There were strong feedback means going down that path and there was an inclination towards constraining things as much as possible.

F

So that's an heads-up I think, secondarily, I think Patrick was kind of exploring this idea of. Like can all these things we don't know where SSH I think we did discuss this briefly, which is that container, potentially getting some kind SSH access to the host and essentially invoking the commands there as needed.

B

F

Doesn't alternative and probably moved on.

B

Yeah I mean SSH and when I are both viable alternatives, but you're basically story you need to store secrets per machine in a kubernetes secret or make them all the same. And so you end up with a secret. That's I would say weakly guarded, probably, and so I'm not I mean it.

B

Might it's different I'm not super clear on whether or not it's better, but it from a maintainability standpoint that might require less custom code than an alternative likelike when s, but but as Gabe just mentioned, the keepalive might actually be pretty handy in terms of making sure things are shut down. Whereas if you launch and terminate your SSH session, it could still run forever.

A

Yeah I think the combination of the Kiba live pod security policies and possibly reducing the image either whether we talk to Rancher and they enable that or we basically fork it and add that capability ourselves, it's probably gonna, be the only way you can pass a security review.

F

So, as an alternative has it been explored whether we can have something like an HMS proxy we're just the H&S calls that are required by the sea and I'd even say that, if expose through a proxy instead of allowing a much wider scope of actions like being able to launch an executable, which would be it.

C

C

How it were like, how would flannel run at that point like you, can do all the asian s work, but like the flannel binary itself,.

G

C

That, just like you are you saying, like do the a-10s proxying through the container and then the flannel is running on the host yourself right.

F

So basically flannel runs in say the CNI demon said and the demon set binds in the name pipe, and so all the H and s related API calls that needs to be that are privileged, are essentially sent down this name pipe into, say. A CNI proxy that executes sit on behalf of the CNI demon set I.

B

Mean I think that's definitely feasible I'm, just not aware of any. You have any code base out there. That's you know ready to use like that. It looked like when s was sort of going down that direction based on the amount of H and s, calls that were proxied but I'm not super clear, whether or not that completes the use case or if it was just something they did for a specific purpose.

B

So if someone has a code base, they could donate. That would be awesome, but I'm I haven't heard of one being open source. Yet.

B

A

I have one more item on this. A little bit related, so I know that gabon been you guys, worked a little bit on the clustered API and enabling that, through the work you guys did with cube, ATM and I think most of the work you guys did was around enabling this for AWS as class trippy as maturing for for other public cloud providers, either. Very specifically, I wanted to know who's who's on point to make sure that windows with clustered API works well for I, sure and I know Patrick.

A

If you guys have someone on on the team, that's gonna look into that from the SE guys. Oh.

A

E

A

So, okay, so so Marcus can I.

B

Yeah, mark and I have been talking to our manager, trying to figure out basically trying to get some stuff shifted around to make room for that. Okay and then I was.

A

Gonna ask for Google for GCP and all the p30 song. So that's my next question.

G

So we we on the windows, kubernetes team haven't really taken any serious look at cluster API and I. Don't actually know the people who are working on that here. So it's not on my radar. Okay,.

A

I will look to see if there's anybody from the community that's looking to into that and obviously surface that up as well. But it looks like, as you probably be first and then sorry, AWS will be first in Asia and then obviously after you GCP and other providers and.

B

So for the AWS is that what you haven't been working on yeah.

A

A

A

Don't have anything else for today. Any issues concerns anybody else, anything that.

A

Anybody wants to bring up, I know mark you, you identified one issue with some: it's a performance issue, the slowness into making the stats performance. I know you want to talk a little about that yeah.

E

We get into the chat we're for Windows whenever either the metrics endpoint in stat summary endpoint gets hit if you're running multiple containers it takes a while to the point order, calls to like metric server or Prometheus starts a timeout with the default 10. Second timeout I tracked it down to a call to the docker on API for getting individual container stats, and it looks like that's taking about 2 seconds per container, and so, if you have a lot of containers running, you can make those calls take a long time on in the issue.

E

I think I mentioned I think we could probably introduce some in-memory caching around that individual call. Anybody has any other ideas or concerns about that. I mean our come Indonesia.

A

My question, and that was that, why are we making the calls down to the docker API in such freedom, the data changing and if we reduce memory caching, are we gonna lose some of the fidelity in the data.

E

Yeah I'm not sure about that, but it looks like these calls are meant to return very quickly, so we're actually like right now with that call taking so long, we just lose all metrics data, usually, which is that unfortunate, so caching that didn't having that one particular equation be less up-to-date seems like a better or like a good alternative.

B

And just to clarify on this, so you say that it happens. Every time, stats, summary or / metrics are hit. It's a boat you're saying both of those fundal down to the same code path.

E

E

Yeah, okay, yeah.

B

So that means this breaks both Prometheus and metric server. At the same time,.

E

E

Prometheus on a coaster and I, slowly, I had a pretty decent sized node and I slowly. Added more just is containers that weren't serving anything and after I got to three containers or pods running for that Prometheus started hitting its great time out of 10 seconds trying to get metrics for these nodes, so they just went blank and then, when I updated, those great time out, I could see. I could started getting stats and the like.

E

This call is where the bottleneck is- and we call this like this two second delays per container per time- that the apartment gets hit.

B

And and I'm just on the metric server side, the default time out of 60 seconds and I'm, hitting that between 10 and 15 containers, even with no load. So.

E

Yeah another thing is, it looks like looks like other other code. Pest call into hds, gen2, get container stats, and that is much thicker. I haven't looked at. How much refactor like I think that if we were to update this code path to college this, and that could be a much bigger than a factor of how these calls are performed. So that could be another approach, but I have been just costed that yet.

A

I mean I think that the first question like to answer is: if there's a way that you can make sure you don't lose some of the fidelity in the data. If we cache it and we don't have them updated every two seconds, I'm looking to what the consumers are, if that's not the case, if that's basically not losing much, then I think Hashem might be the easiest way to to solve this, and I'm I'm in favor of that, obviously, is the refactoring.

A

If you think that we can net significant benefits by refactoring that my beer, an option is walk yeah.

E

I'll take a look at that that would affect some like a lot of UNIX code paths too so and just in general restructuring of how about that calls indicating, although stands for the metrics. In summary, so.

A

You said: what's faster is HTS, you call right not ducking yes, but.

B

We don't have a working prototype of that yet well like today. That's how we get the network metrics and those don't have this latency problem, but we don't know that making the same call to get CPU or memory might not take that long through HTS, like we're still figuring out the next step. After this, like now that we've identified the API call it's taking too long we're trying to see what what we can do.

E

B

Another thing is: I see that it's calling get container stats on just a specific container ID. Is it pasta.

B

Reduce the latency that way by just retrieving all containers, yeah.

E

I'll look into that I. Don't think there is to get the stats for all the containers at once, but I'll look at that. Okay.

E

Yeah and keep all do those investigations and I'll just keep adding to this year. Thank.

A

You mark and and quick note, it looks like who might still get two sessions for Kubik on EU I know they told us, you only accept one get rid of negotiation. I think may have worked out and might get both of our sessions accepted. So stay tuned, I'll. Let you guys know it in the next few days.

A

Well, alright! Well we're out of time! Thank you all for attending. We appreciate all your work. You guys are doing and have a good rest of your day. Happy Chinese, New Year as well.