Kubernetes SIG Scalability, 13 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2017-04-13 Kubernetes SIG Scaling - Weekly Meeting

Description

2017-04-13 Kubernetes SIG Scaling - Weekly Meeting

A

Okay, this is video April, 13th grade scale, meeting I'm David Watson filling in for ball boys, and so let's go ahead and it started.

A

So before we begin there's only one I don't want me agenda. Did anyone have anything else? You wanted to add I. Think.

B

They're the only thing that I have at the PSAs, I'm going to start prototyping, simple contrasting stuffs and the scheduler probably be the first victim because there's synergy there, so you might from see some PRS I, don't know it's most likely going to be wrapping of open tracing work. You know so that way, it's kind of extracted into a library that we can leverage and then you know very simple, rudimentary things they'll probably be added for folks to take care with overtime.

A

C

Do you happen in time line cuz like and like what kind of things if you want to get some.

B

Wow I'm going to start tinkering next week and I will probably have instrument some aspects of the scheduler. Maybe you know week or two after primarily just trying to get sort of tracing one gain experience with it. Right two is to start measuring things like priority and predicates right.

D

B

That way, as we evaluate it, we can have tracing and center I do.

C

That are you aware of the war or jogging around the scheduler performance yeah. That's.

B

The same stuff, okay,.

A

Okay, um so the only item we have is a review of the skate SLO.

A

It some updates I had one thing: I wanted to talk about regarding the PR, and that was the high availability configuration.

A

So I talked about about this originally I had some concern about being specific and the reason was I didn't want to get into the base about like whether we use kooban men or whether or not a particular size is large enough. Instead, I thought it might be more interesting just to collect data, but he pointed out that by definition, service level objectives has to be agreed upon. There has to be some way to agree on what we're measuring so.

A

So I started looking at define a NHA configuration and I started just with what we do samsung with k2, and we basically have five ft denotes 5s to the event nodes and five masternodes. So that's 15 notes for a default control plane and if you look at it how the percentage of the clusters that ends up being or say, like 100 notes, cluster, that's fifteen percent of your cluster, hey! It's not very big.

C

At muchings problem.

A

Here from right, so I just wanted to get especially Timothy's you on what is a. What do you think is a good H, a configuration we can agree upon, should my view would be to go towards the extreme and use 15 notes where the idea is, you could have one planned outage and unplanned outage and remain up operational.

B

A

B

This point like and because there's so many different flavors of high availability, I'm inclined to just rally around the test and say that upstream eventually, we'll have to report this for some blessed version of high availability like and that will be whatever is eventually propagating into the up scripts that people support right and currently there is like one configuration, that's their right and it's kind of weird, but the eventually over time, as could be the end, becomes more popular and becomes the de facto means by which people stand up.

B

Clusters will probably leverage the configuration at Canadian supplies when working on a proposal for initial draft high availability for Canadian, but I, don't think it's a requirement for the testing at this point. I think it's just you know. Marek can proceed happily creating the tests and it's just a matter of reporting, consistent reporting for the two different experiments right.

A

Sure uh and wouldn't change based on whether they were with I availability test or not I. Don't.

B

Pick some would.

A

It need to SLO this change, not possible.

C

Okay, like especially if we had it will have quantum leads, it's possible that they will be talk slower well.

B

Then a part of my effort around the updated high availability design is to eliminate the notion of corn, reads and then eliminate the notion of horizontally scalable API servers goods farce, because if you quorum reads, it's just ridiculous stripe. It limits the entire objective of us doing all these optimizations.

C

I'm not sure I agree I quorum with like I.

C

Did we like it would not call column rates? You can get some really weird behavior. Instead.

B

Because we give a reason we have quorum reads is because people are accessing many API servers to the back end right.

C

Well! Yes, but that's what like 44 min Sonntag on Alpha right, but that kind of how their choice would be used like who wants a chai also for hire four months of reeds. Basically,.

B

We're so like I'll put this in my design, doc to enough I'll, see you see this group, but like the notion that we have, because we, because openshift, is a great example here, where they put back in, they put the hard requirement of coronary talent because they had to force consistent breed you've eliminated the notion of horizontally scalable API servers, you just you pretty much you you by forcing yourself to do one thing to you: you shot yourself the other way right and as a result, you know it makes no sense to have a deployment model.

B

When you know you can have an active, passive API, so ever in a single API server, as of today, is completely scalable enough for most people right.

B

A

So when you say for most people with it, what scale are we talking about I, I.

B

Don't see the bend of the curve upwards of thousands right like I, don't see a problem like we.

C

Are on Sango master for 5,000 pounds.

A

Of the cpu utilization and the reutilization to that.

C

We invited all of your muscles out I.

E

Don't know if I think we didn't publish it officially anywhere but like we are using a single 64 for vm, with like 250 gigs of ram, but memory is not a problem at all. I think we are way below 100 gigs, so it's like less than fifty percent of utilization in terms of memory for cpu. We are somewhere around seventy percent or something like that. Community 40-something course do.

C

You remember to mark skyline 5000 foil, yes,.

E

C

I can just give it a guy in a second once applause yeah.

B

Even on I mean it was a different story back like a year ago, almost exactly a year ago, but now with like proto encoding and gr pc and all the other optimizations, including the caching I, mean there's. There are so many layers now the optimizations that the two having horizontally scale API service, no longer a problem. A single API server could be set to active, passive and you'd, be happy as a clam.

A

So so I can do a preview to the day that I mean unless you, if it actually does scales to say, 5000 those there's no reason to have more than a single master. Well,.

C

It might be needed formula for hire by the regional sports performance yeah.

B

I mean but daily passes. This is loading a second.

A

Right, what about for STD servers, let's via this.

E

So again, like I think that having faster that CD is like for high availability, not for performance, because performance, obviously like not better with custard at CD, we didn't do recent experiments after switching to a CD, be free, if really need the second instance of four events. It definitely was that was the case like in the past, but it was like year and a half ago, or something like that. So we probably need to verify it again if it has right.

B

Ideally I'd like to eliminate that from the deployment. If we could, you know, as a default configuration I have.

C

Some like that, like 45 thousand notes, we like a p.I server, uses around 23 cars and like a candidate around and well control them another like a loss.

C

For the manager rounds by to actually surprisingly low and littler the same, like that's like some percentile, which of course I if you schedule a lot of things in the same time, the scheduler timber nice arbitrary amount of time. Oh.

C

D

Please look I settles in here. I put the issues, they thought it was solved, we're not sure it's 100 pence all did we just cuz the API servers scales.

B

And was taking.

D

A ball you take enough all of its own resources and over there still weird issues that you can see in other places like you know, and I'm, not sure how you solve this, but like in other places like in terms of dropping breaking HTTP connections and stuff, like that, like a slight good stuff like that, I, don't know if you need you guys, look at this again. It's tough! This.

B

Sounds like the crap panic handlers. This doesn't look like the this was supposed to be eliminated, but I know Jordan added them back like the.

B

There is a bunch of like I, don't know in the one x series there yet there's a whole bunch of panic handlers at it and then they're added by Jordan and then daniel said, like we need to eliminate all the panic in which we should just feel fast right and because you know we burn what happens is when you have a panic handler, you're, basically masking actual issue that occurs in which she just going to fix those issues were possible and them yes,.

D

So when you turn really crash, true on that was the experiment.

B

D

Always plan within it, you can reproduce this stuff like that, otherwise.

E

D

It gets swallowed, inspected, screen, it's a really weird issue and I. Don't know what makes specific to the openstack openshift overlay stuff. We have going on whether it's I mean obviously since Daniel and some other people were working on. It is not specific to yeah.

B

But because it doesn't really relate to the performance aspect of things, it has to do with, like just general behavior, if its proper or another yeah.

D

I, don't know how many p I services running on to and I'm what I'm wondering is whether or not this ties into whether or not you know me: I have a housing available, API server space maybe still want them to prick. Maybe maybe even the API server can handle.

E

A certain scale, maybe.

D

Not that doesn't necessarily mean that it should.

D

That's a tricky question: I, don't know the answer to it. Well,.

B

The problem is the whole active active farce that I could just call it a farce, because it's it's a lie. The further you dig into it, because if you're forcing quorum reads down into the back end of things, it's basically saying you're, not active, active okay, um because you're, not stateless anymore, so I think you can do either or but I think the panic should be cleaned up and handle right. We should be panicking you're.

D

Just going to say you just swing so what you're saying is they're using to slow things down of a different place, absolutely.

B

No, it's not going to slow things down at all you're going to speed it up because you're, but you're basically saying like you're going to take. We.

D

In a quorum radios, you know sterilizer the different legislative quorum.

D

B

It's going to force a to read, yeah.

E

B

Reed across your back in right now.

D

D

Just wanted to put it on the.

E

Radar, if anybody knows anything about.

D

That, let us know I, know I, know Connors looking at it to you. If.

B

There's panics happening in the API server that the.

D

Hack solution that we came up with was dialing, Max and flight down. It was.

E

Really a thing because it.

D

Can handle higher max in flight, which was something that we talked about in the last six scale. We should start auto-tuning that, but the reality is auto tuning is tricky because even if you can handle higher in flight, other things might break when you actually have large amounts of pods, but you're.

B

You're you're talking about a specific instance, though, like we need more detail and what the panic sorrow that are occurring you my very well again.

D

I, don't have a full simple producing to see if I can come up the parts.

B

So it's a state to go back to the H, a question in general, like I, think so long as the tests are consistent and report consistent. It's irrelevant, and we can just say like eventually upstream will report both a and B for both configurations once be, is fully supported in.

A

Like the orders you mean by kuba or whatever, the whatever.

B

The de facto way is by that time right now today, it's still good luck. Okay,.

C

Yeah well, there's also another discussion about this dog back like where should it live outside in other sink directories? There's pretty much nothing I, just only can't go out like meetings and that's it I kind of don't know where this kind of documents sleep or should live or die. Woord belongs the.

B

Community holds all the other stuff right like so look the only place I can think of where it should be. Nice.

C

Yeah I could be communicative like, unlike from the design proposals or something it is not a design proposal.

B

Here but like I again I, don't I don't have an answer for where else it could move.

C

B

Know, that's like a you know: III, don't want to say, go to the PM group because I shake my fist at them, but uh that they them or the contributor working experience group might be the two places to ask.

C

Okay, well, no.

A

So I don't know, I feel a little bit uncomfortable just saying that word we're not going to decide I.

A

Feel like we either after, like pick an arbitrary or say we're not going to pick and since we're going to pick, it doesn't really matter what we think we can change it later, but we have to have something. That's a brief on right. I definition! That's what solo is yeah.

B

But there's not, there isn't agree upon one so I mean like until there is such time as they agreed upon and high available deployment, which there isn't one right. We just had it just left as a to do in the document business. For me, that's fine.

C

Yes, I right here: d, on the other hand, like also I, don't think we have at least we will not have capacity to just test. Everything can work so just like want to start with single muscle deployment. This is the thing that we're running and if someone wants to pick up and create like arbitrary H, a deployment and trung tests, I'm fine with that, because I dislike a lot can change like this is not said something such as I, especially environment.

C

I sped them and I expected an environment into which we measure it will change a lot like for sale, for example I if we increase the density allowed them city, so on a large number of part in a cluster, the probably it will change and southern stuff might change as well like some type of things like marks and flies or racing with.

D

A person, clients or watch on Oh.

C

Even if someone will just pick something called bichara for now and Ike will dedicate some resource afoot, I think I'm of 4, I'm also down so.

A

I totally agree, I think the SLO I mean from this meeting alone there at least two things that I may want to change. In case you for Raj configuration both the number of API servers and whether we split CD events or not these products. To be brief, so the important thing is to be right about the configuration it's just to revisit arbitrary and then write it down and make it a goal that we strive for it. So all sorts of all and see if we can pick up some of that testing.

A

Yeah I, guess I think the important thing is we start collecting at least cheap just like they feel. Like one data point it's useful to, we can see at least beginning of a spectrum of performance. We can expect.

C

Like we will have I cud single master plan, H I, if you are going to do it and bike and I think like I, don't we can compare those stories like that they will have completely different like everything, but that's fine. I couldn't need to because I think that the SLO is just with oh such a lot adjust to these people. Some ideas like or some reasonable, like show them what's possible like how we tested and just make everyone aware like.

C

How does it look like and what we set ups were running on so light in the canoes the our result, if they want to.

C

See I, don't think like that. I visited this, as you mentioned like I did. This is that setting some blessed consideration. It's saying that when, if you're running this kind of configuration, this our dealership said expect for those tests all right and like when you look at this this way like actually well it's better. If the configuration is like some reasonable excuse is reasonable and anyone can relate to it so designing something close to it. But if it's not then well like nothing is really bad about it. Right.

A

We'll just change it later, as this variable and.

C

That's like it will cause it to closer to the decision for HIV. That looks like great.

C

But, on the other hand, are a great team, like I don't think we should block on it like if someone will pick it up like like they will just kill, the dog is better to do it out there. Yes,.

B

Exactly so, as only the duck just has a little titty there. That's that's all that really matters, I think from the high level perspective and then over time it will be blessed. There will be a blast configuration right or four kooba or whatever that, whatever the standard way of deploying things will be right, but that that'll still occur. That'll take a little bit of time right right.

A

And that's that's been my concern when I've talked to both aaron and all on dro is that I think it causes the time it takes the private of less configuration. We should walk on it. Instead, you should wear. Resources, exist, collect data well,.

B

There's nothing that prevents you two from running your experiments and then you know thinking in the sig like saying hey, you know, merrick your results. You got so much different than mine and talk about it here. That's the purpose of the same right and.

C

Like this is also like what Alejandro's doing like, we need to understand what are the difference between checkups, except, of course, they're, like holders and allow whatever you running.

A

Sure I mean for the purposes of the documents. I think it is important to to be specific, even I agreed a specific just to do.

A

We haven't been specific, yet that's fine too, but I don't think we should wait and tell us something blessed before we become specific I think we should wait till we have data like we had right now that showed a 15 notes, control plane achieved the same Ethel oses, a single master configuration then I would feel comfortable, saying to be definite fellow for both single NHA configurations, because that's an extreme case right. We have as many.

A

Control plane nodes is when my reasonable one and then, if you, if you reduce the number, I would only expect things to get better, in which case we might change the slow to say to you only idiot fewer notes.

A

C

Well, I think that's it's not about having data about someone that again increases for doing it.

C

E

C

Let to do is kind of asking for help I if you're interested in how it looks like and are willing to run those tests just fill it up and run your test. Babe.

A

Okay, we have two and a half minutes. Does anyone have anything else? You.

D

Can talk about the tracing stuff at all? Did you too I did.

B

A little bit I, basically sending this start on a pretty soon and I'll, see see the sig on PRS that go through I'm, gonna start with, probably without the utility and just very simple, trace stuff. First, if.

D

You ever were we're super interested that that logs is related. Do you guys want to have a unified front, or is it possible for us to have a unified front on how we feel about logs? Because it's you know there's like when ppl on earth test anything that large scale to think about logs completely differently and I? Think in our testing thing with large scale, it's like these two different camps and anything that.

B

Sets shown its head several times now, where I think that logs in general, if they could be, you know to the point where they could be either n or n P that they shouldn't they should be at that rate, because it's just you're gonna get so much spam in your logs that it's almost impossible. Yeah.

D

I mean I, it's like well I mean. Can we agree as a cig that we consider log with grow linearly with respect up either pod for notes mugs? That would.

B

Really help with I think you could use that as use case and CC the fig on it because, like I know, David Wannstedt, yeah, David Oppenheimer wants to add that one log and I'm like this is going to be so goddamn spammy. If you add that log um and you're at you know, I didn't know like 10,000 or 60,000 pods. Every time a pod doesn't fit you're going to have yeah. You know yeah yeah.

D

Even if it's just for the first time, it's gonna be also edit. So what it is it's.

E

About agreeing that.

D

Logs are not going to be useful for this. It's just not going to be that way after a certain scale and that's good even on a term node cluster, but.

C

I kind of don't agree like in Cronulla, like we got the like them, nor what amount of love and you actually do warn those logs, because, like one something blades, you need them to secret right. Well,.

D

That's why the open tracing stuff that Tim is working on is so important and exciting is that it will allow completely dynamic forensics on any part of the system, and it's actually something that you don't have to write.

D

Custom tools to go grew up around for crap aggregate stuff, and you know I mean I, understand the need to trace things and I understand you need to roll things up and figure out why things went wrong, but coupling that to the logs means that it's always going to break down once you go, sir, in this scale and well after certain skill, you're, just not going to use that stuff anyway, so you need to have a redundant solution anyways. Why not just use.

C

D

Networks really.

C

Know like there's like a number of tools that can follow you today to mind a log right and signature betray not among them yeah.

D

There are just there's definitely a lot of tooling come to mind logs, but do you want everybody to need all that tooling I think.

C

I think that when you don't have it like you're doing something wrong, there's.

B

A certain level of jurisprudence right, like you know, obviously, if it's NP in its hot looped, you know we should not be doing yes, yeah.

C

Right when I saw.

B

Like our boss, yeah, and if you you know if they're super linear in any case, that's also bad right like the whether or not you know it's directly proportional to like you know, the number of pogs number it knows seems completely fine. If it's directly proportional to the number of todd's. You start getting a weird case like.

C

What I directly proportional percent, or just there to like you, are at games having like one law like whatever like number of not perfect, you can be insensible we're against the idea.

D

At any time, you're losing a shin by not logging it. You they'll force that information to be in the logs like and that there should be some cut off and I think the l event is that's the one that I would like, but more than he. I guess you know, yeah.

B

I, don't really have strong feelings on David's issue, honestly, but I think as long as it's not hot looped, you know NP, then I'm I think that's like the generative case, where you have to fix it. I don't have strong opinions on dancing and originally I push back against it then I'm like I, don't care, you know if you really want it that's fine on, but we already have the data right. The data already exists in the pond status. So even.

D

Posted stuff yeah still the one loss for every single particle system; okay, so.

C

Yeah I think what immediately might be me like one lockeford guys in some amount of time per pod or just like when you start it or when you delete it or whatever yeah.

D

C

Liked it if every time, plaster and you'll have like one block per spotlight over like a week like this, a one love for pot over week, I like if you just law like if like ID, if you do not have a lot of churn and you write in a cluster- and you write like a log line where she like, when you or whatever it is you or ten or hundred log lines when you create a pot, and you do not have a learn, love Chad, I over text of days like it just is nothing right.

C

D

It gets ugly when you're scheduling you know if you have district it when the thousand pods at once, you got a towel and then you parallel. You got summer schedule. Control like if you got I mean even though scheduler isn't pleased came in for outlets where you've got such a tremendous.

E

D

It needs to be rolled up anyway. The tracing stuff does so that's where it was profound as well, which the coin right.

C

Okay and I kind of don't agree, I like one if something's happening on the system, there should be locked about it like, of course, we can kind of discuss like what level of logging and how much data, and what noise at least progress. Then we all agree that we don't want our venti.

D

We none of us want that. Some of us disagree about whether you can have all of P right, but I mean how we all scream it that we all of NP logs anywhere our bug, regardless. That's I,.

B

Think I think it's funny that we're using n is a node and peas and cons, but anyways get ya. Look a little. It's funny, but yes, I, agree. Yeah.

D

I mean, look, let's see alice, that's all. I know.

B

I have to note here we always.

D

Would be all of them, keeping that I.

A

Think, yes, very on that note, um ticket is rotation which I think is talking about. Logging this week is occurring right now, we're five minutes over. So any last minute, honest, so anyone else. Okay, thanks! Everyone sounds good day.