Kubernetes SIG Scalability, 14 Sep 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2017-09-14 Kubernetes SIG Scaling - Weekly Meeting

Description

2017-09-14 Kubernetes SIG Scaling - Weekly Meeting

A

Okay, there we go so this is the September 14th scalability meeting public, recording and I think the request was for just a quick update on the network s, hello, testing from Tim.

B

Okay, so I think like it wasn't really about testing, because we are still, we still didn't define the SLI Sandnes allows in their area of networking. I started this guy, like I wrote an initial proposal of forest lies, and this allows for the networking area and the dock like isolate the dock with for now on, v6 Cal ability and cig Network and I did quite a lot of feedback on it, like mostly the feedback was about how to actually define them precisely not really about what exactly we would like to measure mostly like.

B

There were some comments, also about like what exactly to measure but like those minority of them like I, can give a quick again proposing the dock did.

A

You know, I was looking here in the notes or the lane. Did you add this onto the existing ones or.

B

Now there is that new dog actually, for it really.

A

If you, if you post it here, I'll put it if you post it in chat here, I'll put it in the notes or, if you can add it to the notes, sure.

B

Ok, I posted so basically like the SLS I'm proposing one is like for like I, don't think it makes sense to really go into like deep details, how exactly they will be defined, but it because it seems like it's. There are some some back and forth on its, but like the high level, like the things that we would like to measure is like first like it ends here of DNS lookups like lookups to DNS and plate and see for them.

B

The second one is like about the how fresh the day DNS is serving so basically like the little latency of propagation like like, basically how fast a new newly added back-end buying some some service is like also served as a like there's my DNS. The third thing is.

B

Latency of propagating changes to say to be slow, bouncer by mekinese, the basically like how fast from like when Bob, is getting ready or not ready or deleted, or or something like that. How fast it's this like is actually reflected in IP tables or whatever else mechanisms we are using for the internal load, balancing.

B

There is also like for that something that the SLI that is really cloud to provide their specific. So probably there will be no SLO around it like how fast we can create like external load balancers, just for basically know that how fast it is, it's not like really about enforcing anything, and the last thing is like basically, the over ahead of kubernetes and the high level. It's like how much light and c-23 quests is added by like all this IP table stuff and all those concepts that are like burnett is specific.

B

Comparing to your machine to machine like like, but that's like very high level. The exact details are still being discussed like what exactly and how exactly measure and stuff like that. So maybe we can connect in like.

A

Okay, yeah I was just trying to look at the dock and I was having to where request access. He probably need to keep an eye on your keep an eye on your dock permission or no I.

B

It's currently shared with guys it's gotta, be needy and seek network, but yeah. Okay.

B

So probably, if you log us know.

A

If I'm Argentine, it'll.

B

A

Work if I'm logged in properly anything.

B

But yeah I already shared, like you, gave an access to you.

A

Okay, very good, so I made a note. I made a couple of high-level notes here and I'll get the I'll get the doc in the notes as well. What's um is there something specifically motivating motivating the motivating the networking work or so so.

B

Basically, we know that there are some like with, for example, with many number of services that, like iptables, won't work. We already know it and we would like to more formalize like at work at what scale and what what expected, but what users can expect from like from networking stuff inquiry, is currently it's like in theory like we can, networking can be totally down and still like our current SLS will be satisfied, which is just like perfectly right.

B

It's I would say it's like far from perfect on.

A

The VM, the alternative, would have been too focused on I, don't know, Clayton Clayton likes to show up here and talk about running tens of thousands of name spaces. Things like that.

A

That may be a more narrow use case, more specific to the Red Hat, the Red Hat to the open shift method of operation. So.

B

Yeah yeah I think like both are something like I've heard from customers personality that they were using thousands of or even tens of thousands of services, and it doesn't work for them, so it probably depends or they would like to so. It probably depends like on what customers we actually look. So I think that both are you important and the true which one is like more important. So.

A

Your your your, the current current view is that it's more likely at the typical kinds of scale that networking is getting slower and slower and having scaling issues so it's time to wrap it with some more specific scale testing to ensure that we're catching regressions generally.

B

I think it's not really like. It is also getting slower and slower, but there's some threshold at which, like iptables, stop work pretty much at all, which means technically, is basically almost down then ready.

A

This is the wandering into IPPs itvs territory with iptables equator. Is that.

B

Yes, I think we we start like. I think I pv s is even in alpha in one night, I'm not hundred percent sure, but I think that I'm sure that the code is already there and I think they are going to announce alpha for it right. So if.

C

B

We didn't do any specific, like load testing of it, yet because it's pretty fresh- and there are still some some things that needs to be fixed in order to make it really more scalable that we did for IP tables. But, yes, I, think it's radially.

A

B

Are we going going to go this way, but.

A

These are suppose would be critical to defining whether it's broken or not. Yes,.

B

A

So, okay, all right! That makes a lot of sense.

A

I'll make some I'll make another couple of notes here. So anything else, a ladder by light I see it looks like looks like Aaron, is putting stuff testing stuff in a nun tree I'm sure he won't say something.

C

Our scalability tests, so you need an agenda for the meeting. I have linked the GCE scale performance. If you see these scale correctness, jobs in the release master in Test cricket dashboard, they both continually been failing. I found that shawn has created an umbrella issue to track all the issues related to the bank correctness. Failing tests, yep, that's right. Okay,.

D

C

Other thing there are no release winning scale fests so any event. The code thaw happens. We stop looking at the master branch and we start looking at the release, one a branch or arcing.

C

So if we need to be continually looking at scalability as.

A

A signal forgive.

C

Them go we'll need to make sure there are at least one and equipments of these jobs, but so, let's start with what's going on with the master jobs, I.

D

Don't think our scaredy-dog played the 5000 per phones and correctness are actually running against the 1.8 branch they're running just against the master and yeah and like for the correctness, like you told, have filed a few issues and some of them like three or four of them got fixed last week and there's.

A

D

A few which there's still a few failing ones and the thing is some of them- are actually flaking like a few tests are actually flaking like they're they've passed once and they are not passing again like I did not file issues for them yet because they, like those very sure, those two is which, which passed at least once and so like and I, think the correctness is kind of in a good shape. The correctness tests, but yeah we're having a real having regressions in the performance ones like the density. The density test has been failing.

D

We, the load test itself passed passing, but density test has been failing for few runs already and like. We recently figured out one issue, so we actually used even small nodes. We were using given small nodes for the five thousand or cluster and which has like less memory about three and a half GBA or something, and we were seeing a lot of whom even three.

B

Once Mao is like less than 2 gigabytes,.

B

Ok, it's like 3.75 is n1 standard one in oh, yes,.

D

So yeah, so we had less memory on the node and because of which there are rooms and, like booms, kind of end up making the cluster reach of bad state.

D

If it's happening across all nodes, because cubelets and NPD's are at Neutron detectors, are actually sending a lot of events like a huge amount of QPS of events and node status, updates, which pretty much was thrashing the api server and like having high memory usage on the on the master on the api server, and it's eally so like this actually helped increasing the node size to N. One standard won't help because we think it helped with some of the latencies they got better and it also included the resource usages of these components.

D

But this is still one more regression and we had to find out, like I've, been trying to divide it for a couple of days already. You know I'm trying to find out why some of the calls still have high latency, like like prominently. The node status and delete pods are having high late in season, and the too flaky like some calls, are having high latency and some other calls are having high latency as other times. So.

C

Yeah this thing here I mean achieved by one eight: nine, nine, where API colleagues she shot up, alert, Busters and.

D

This one more there's one more which to spot startup latency shutting up, but that seemed to be a flake that happened for one run and didn't happen for the next students, I think the students and, like it's it's fine now the bloodshed I can probably close that issue down like the issue. The problem currently is only with the like, at least with only with the API latencies.

D

C

Yeah, so just being really silly and looking only at the test grade, it looks like more.

A

C

If today's correctness run Tuesday is for at this room- and it seems like correctness- maybe isn't giving us sit over here if you're saying everything was fixed, but all.

D

Was fixed like there was, there were seven flakes which actually sought, and at least students like back when we actually had two proper rounds like which failing both of them and there are much looking at it.

D

And yes, there are a bunch of stem which are failing. Yeah I'll find issues for them. Yes, I, just sorted. Restaurant prepare for networking, yeah.

C

So it's okay that he's running it against master right now, because wherever have the release process master is the authoritative signal must be with please we stopped looking at mastered, start okay opponent, so I asked you to work with sand or somebody to make sure that the one eight versions of these jobs exist. We still need to be using a specter, Nelson cool yep, I'll change it, and if we're.

D

Still seeing this level of failure.

C

By next week, I would ask you, show up to Monday's Bernadotte meeting, because I anticipate that's what we're going to start getting individual states for their release, blocking issues right now and Polynices Herman's blotted, ok, cool right! That's all I has a release representative thanks guys and we're good.

A

It's awesome, it's awesome done from Jason opcodes wish yeah. So.

C

Whoa I haven't reached channel today, but we did yesterday.

C

We thought that things looking good right up until you know is that pretty much all the thing up there tests were failing, and so we believe we've identified one or two fixes that might help with some of the lives upgrade fests and we wanted to get one complete run of great vessels to see if we truly accomplished that or not so we'll make a determination today, whether or not we live code freeze or not excuse right now, yeah right now we're in I'm kicking it down the 24-hour kicking it back for 24 hours.

C

I would probably anticipate if we feel it. We don't have a great signal if we don't do today and we look at it again tomorrow and we're not happy with it- we'll probably leave it frozen over the weekend, but it's too early for them to say for sure right now.

C

A

Well, thanks for me any other any other topics for today.

A

If not I think we can call it off thanks everyone, that's all.