Kubernetes SIG Scalability, 6 Jul 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2017-07-06 Kubernetes SIG Scaling - Weekly Meeting

Description

2017-07-06 Kubernetes SIG Scaling - Weekly Meeting

A

Cuter, okay, hi everybody, it is Thursday July 6th. This is the safe scaleability weekly meeting, which will be posted on to youtube shortly. We publicly recorded meeting and we have publicly available meeting notes. I, didn't see anything on the agenda today, so I added a pluggable event of storage based on the fact there was a mailing list read about it and that's the thing I think dedicate most of our time to you.

A

However, once surveyed a crowd here, if there's anything specifically, we as a sig would like raised during the 1/7 retrospective that is being held tomorrow in two fluids. Maybe we have to say would like to start talking about our roadmap for kubernetes one needs I. Don't think. Today is the day to walk through that list. That I think. Perhaps next we could be really cool.

A

We could have a document or something put together some issues, you know, throw them in the 1/8 mile set and that sort of thing so that some point during the community meeting we could say again here's what we're playing are working on um is there anything. I am missing from the agenda.

A

Laundry machine machine says no: okay, um ken and Tim I will look to you to lead the discussion on huggable bag storage um is.

B

The big Googler who want to talk about it there right. Why don't you start the conversation and then we'll we can banter about thoughts up. We've had over time because they've buried it is upon who's going to commit to what at what time frame. So why don't you go ahead and and I'll learn some new, your thoughts there, okay.

C

So I came across the design some some time ago and I'm not.

C

It just sliced across through the area I'm focusing on right now, and the crossing I would like to be able to retrieve events from long-term storage and in the design. I saw these idea of streaming them like on the on the API server very quickly, so I wanted to well first of all here. If there were any considerations, if we want to receive them in the same way so to have the API able to reach not ultimately TV or but also to some other back-end.

C

These are talking the fear, panic arena so.

D

In this proposal here, I think that the way I've outlined the release is that the it would be just a pure kind of like teeing off to another system and kubernetes doesn't care about what that system is. It would still be using sed for retrieval and the API calls for dippy events for this thing, which still works. That is mostly just to kind of minimize the disruption that this would cause and still allow people to get raw event streams being sent to another system if they so choose. So.

B

What Daniel had proposed and what Daniel is proposing, is to go a step further and to share the API using aggregated API servers which I'm not terribly opposed to, but it's kind of like using a sledgehammer to pounded attack, but it's not a bad idea and I don't know what people have thoughts are about that, because eventually we would like to shard the API itself through a greedy API servers. Released events gives us a way to do that. It's not been vetted but I'm, aware of like the ability to shard the API this way yeah.

B

So it will be blazing the trail and I. Don't know if that's a good thing or a bad thing. Usually it means that somebody's going to walk the pain.

A

um Me part of my concern with that is I personally, don't know what the use cases are on the quote-unquote, critical path or vets always had some people describe them to me as nice to have things that are purely for the developer, to look at work for debugging and troubleshooting purposes, but I'm also under the impression that sometimes they're, the only way you can find out certain information or, if you or want to like ask when something happens, the event is the only way to whenever I've asked like if they're like a consistent, do we have a standard for events?

A

Is there an API or something like no do it? Events are not an API you're, totally, not so I think I would seems like what's lacking from this is the the surface area that we might potentially be impacting, and the reason that this particular design dog is missing. That is because we're just like a wiretap will not actually impact the functionality directly starting with right.

A

E

I are going to a redesign, um so I quickly substantially all right. This is a completely separate effort. I can like I just finished a big designed of today and I, just like gathering some thoughts internally and I will share. Share it. Wider I know this be fine x3, but I expected soonish.

E

It's a long one I could I could redesign of the even thingy and.

E

Yeah, so the end like DD top is like I spoke with Brian about it that once we make like um take this to the events themselves, we shall we and somehow underlies them. We will be able to actually make them an API, which actually means only one thing, that by the values of all the even feels like whatever I, this involves, object or whatnot will be treated as API and will need to follow our duplication policies.

E

B

I look forward to reading the proposal is d. You are you going to short it using aggregation? Are you going to just completely over all the API, etcetera I just.

E

Want to change the not touching the aggregation at all like I. What I want to do is basically remove like compress the all like classical think event, so I can set up attending every deck one event every time like something that sounds a bit like a crashing whatever, then just like send like three events by the first one when it happens for the first time, the second time when it notices that is actually I got something that does loops I and like the thread one when I this loop finishes to kind.

B

E

B

The thousands of events in between so but that that doesn't actually change the fifteen to another system, no.

E

This thread input, but I, guess kind of removes like I get more like depends why we want to te, but as this will remove this trendy deep logic, we will. We currently have with well yeah countin yeah.

D

I think I have so I have a branch that I was working on back in May, at least on at least it work to the proof of concept on moving the event compression logic able to client and into the server and I decide that, if I understand correctly, you're saying that would be kind of one of the prerequisites would be that the client would just stop trying to be intelligent about only sending certain events and duplicating them and instead just send all of them and knowing that logic on the server or they are quite.

E

The up with kind of picado test, blonde sent and I think my tank will be inclusion and like once, and they repeated event because I they actually are not really important, like what user actually need to know. Is that like there is something that constantly ongoing yeah.

B

E

It actually doesn't matter if it's like four hundred thirty third occurrence of this particular thing or 456. One right, I know one care yeah.

D

So I guess we have different.

D

We have different goals in mind here, because my goal would be like the reason that why I wrote this, this proposal was more that if we want to have an external system that can do something intelligent with these events, instead of having the clients say well, this seems like spam, so I'm not going to send it instead of just have said, send everything and then the server can choose to ignore things, and then you can send it to another system that might be doing more intelligence than what we put into the client, because you don't want to lose that information going around.

E

Well, its neighbor can't handle photos. I know that right, I'm.

F

Gary, like.

E

Even though you know not only its detail but like we saw like okay does affect server starts to responses for 29th I, pretty much for everything, but of you. Okay, this is the compass angle. Okay,.

D

Yeah, so this is, these are the kinds of things that I haven't. I haven't worked enough with in person to I've seen so yeah totally.

E

The aunt makes a lot of sense because I identify naturally it because, like even in DHCD like just like epithelial upon hours, thanks drop all this and I quit the key. We can actually tended out to some like resistant storage so like so they will be able I really be available like for a longer time. Yes,.

B

So that's kind of the there's two parts. Then it seems like to two separate problems, both related to events right and the question that what's your name, next America Daniel.

D

B

So the question that Daniel was raising was separate storage and the question that Marik was raising your topic. America was raising was the the extra spamming right and but I think they do need to be treated separately, but there needs to be like a comprehensive view of how events are handled from soup to nuts right.

B

So the I don't have any sir for the comprehensive view and time frame for that, although everyone knows that sucks um I think we can do incremental pieces if Google's going to commit to the spam cleanup and long term timing, you know we, if you're going to modify it to be like a supportable API. Then we need to review that to make sure that it meets the needs of treatability right yeah, because I can't do tracing it. Yeah.

E

Like we can I believe you can come in community to like figure out this bomb thingy right as I said. I will send this to sender design next week and like, of course, it can take each other long, Patrol write this but like if it will get through like fast enough for it to be implemented in one night, and we will do it and afterwards, I cool just need to like all the windows over also given to the customer in Christ sing around now.

F

E

With that, we'll need to help wind help good from pretty much everyone from every single component of Iran yep.

B

We started digging in this one of those pieces, but some of them have met resistance, so I say resistance was about spamming, so I think. If we had a solution to spamming it would it would ease the concern, because much of the additions for the modifications to events were just to add more granularity and visibility into when I say more visibility, not granularity into what was actually going on right because we have. We have events that are just actually useless, expand the hell on the system. So here.

F

D

Cubed John, it's something like seven or eight events, just launching a pod like every time a container restarts it's really good owner all.

E

Of them are kind of important if you want to try- which is sad. But yes, that's been even more because, like I think they meet Evan event per container class, one or two events per volume, plus whatever Provine and containers for the separate. So.

A

This kind of maybe another orthogonal concern I, don't know back to events as API right now. Events are not considered part of the to performance of the kubernetes cluster right. What.

E

Depends what do we mean by that? I got two levels of what blooming treated by not an API. One is events, are an IP I objects and they are treated as an API. Only so like we can't just get rid of events, and just like, whatever recorded.

A

Which specific events happen in response to which things at which time are not part of the conformance tests at define whether or not a kubernetes cluster is a kubernetes, so it is possible for them so I bumped into this a while ago, in the implementation of a third-party scheduler. Can you still have that third startup party scheduler be considered a conformance scheduler if it doesn't fire the exact same sequence of events that the scheduler self pipes here like I've been alternative to?

A

But if it doesn't fire off that sequence at seven or eight events right that certain book is not covered at all by tests and if we get, if there's a ten on offense, you probably ought to again I kind of go back to be great, to see some of the use cases where vets are necessary, like you're, just selling out.

E

Well, if we want to be able to trade the system in a sense using even.

A

Right I kind of goes back to well. It makes it easier to debug but I'd like to understand, which are the specific event flows that have either right. I'm, saying that we use Kanzi.

E

Which means I like stopping the pod and then she scheduled to go away yeah.

A

I my thing is like I'm, not sure if there's a document said anywhere and I'm, not sure sure that sequence is verified anywhere right. So if for happens to be in the system, the system itself is the specification seems a little loosey goosey to me. That's.

E

True but like amount, even Titans, viewable.

B

The system is super loosey-goosey, so underspecified and implicit that it's hard for me to even put words around at their colorful metaphors. The usually.

D

People use them as glorified like debug logging, some plans they just like printf things they just like. Let me just put an event here and invent here.

B

Ideally, they would show pods lifecycle changes if we had an actual state machine and they would be the transition pieces on that state machine right, because that way, you'd see the entire state transition, as it helps around the system, because we can't do that with anything else. That's are the only mechanism by which we can use to do this. We can't use tracing, doesn't work.

A

And the other thing like I totally understand this is a problem, but this seems like the same repetition of we're doing all of our rate-limiting client.

D

Side and not that's.

A

Being implemented back pressuring the server level, which means we have to trust that every client is a good actor there's.

B

Back pressure, but yes, the back pressure is basically I. Give it up for point.

D

A

Right beautiful that pressure, I guess I, should say: predictable, tunable back pressure.

B

There needs to be a mixture of both so I look forward to reading Marik's proposal. Daniel I, don't know exactly what you wanted to do, or what your thoughts are right other than you. You want long-term storage, you want to tee it off and we also want the same thing. The question is to what and how I guess to how is more important at this point, because the what is just the events the?

B

How is going to be like whether or not we do some type of team mechanism that can is proposed, the webhook mechanism or the sharted api server daniel seems to be very pushing tour, sharted, API server right this.

E

Will be also Department, it's the hardest mass, but there's no one walked in here. Yeah.

B

Well, I think I think long-term. It provides certain flexibility very good.

E

What is different, okay, I! Need. You keep on engagement, looking you think that appears in one second, like no an ultimate work.

B

Well, red has tested it because they need it.

E

When you look like in reveal the code that will be written on it,.

B

I think it's David eats would be the reviewer actually.

B

So I don't know Daniel did you have thoughts there? Are you proposing, obviously, the the idea of teeing it to Kafka? Q is the most common thing right. So are you planning on executing on this in the one eight cycle to.

A

B

With the sharded API server so.

C

We will start some prototyping, but not something that will be released and it turns out to be ah well. We don't know yet whether whether it's feasible or not, to just spend like what what amount of depends. It doesn't make sense to ashram, won't encourage just like that.

C

For now we are doing some prototype and I just wanted to.

C

Keep track of what's happening around events, because if everything keeps changing and quickly made, my may become obsolete whatever.

F

C

Have fertilized.

A

He there's AI mind some scoping stuff heard something to maybe help us scope like what I would like to see is a concrete action item. Out of this is we are willing to do in 1/8 and what the roadmap is for the rest of its 2017. To accomplish this to protect 82, it key dates. The heat time for 180 are by August 1st.

A

We should have a feature in describe to what we want done for 1/8 by September 1st code freeze happens at least that's what the proposal is right now, that's basically like a month to decide if we want anything committed for 1/8 and then another month, actually really really doing so.

B

All I can commit to for AP, for events is basically review of proposals and potentially modification helping to create a proposal. That's all I can commit to because I got too much other stuff on my plate on the committee inside of the world so and.

E

What about can end? The team ahem I mean.

D

So it seems like these are conflicting ideas that you and I have that, because mine would be more to just have the cubelets end or what the client is send. Everything and, let's ever deal with it in yours, seems to be going but more intelligence in the clients, so I don't think we should both be pursuing these. At the same time, separates.

E

Like completely separate, thank you exercise, one is by protecting API server and it's again whatnot and second is putting human simple non samsara. Try like it's not constricting at all. I'm.

A

Interpreting it as marriage, changing the content and you're proposing a mechanism for where to shovel the content to well.

D

I guess the reason why I'm seeing this absolutely discomfort here is that I don't think that we should. It should be lossy right if we're going to be teeing an optimizer system, it should be. It may be that that system doesn't want the client to second guess which events this system might want or might not want right.

E

Well, maybe we're changing some on date, so you want to be lossy. There won't be any that implication. First half this will be all the events that's it can generate. It does on the every single iteration because they could not see forever. It.

B

Just won't be stupid.

F

D

I made miss wide, really helpful to read the proposal because I, you know, there's a lot of nuance here: I think yeah.

A

Okay, I'm very disciplined. Sometimes you don't have a concrete example right, but sometimes you might be able to derive useful information from the frequency of events, not just account other than yes.

D

It's that's basically what I'm trying to get at I doesn't just account, might not be enough. There.

E

Isn't anything disability that we just conquer as we penguin see? This is the computable. It won't happen even.

D

If they don't go to a city and the raw creativity.

E

Even if they don't like eating API server and I, not even.

D

If they're not hitting the API server, videos.

E

Not you think I do not fine, but I'm. Everyone.

B

Can't do that? Okay, you need to pass the client certs, allow the crap ball and Eckles and endpoints to every single node and no one's going to allow that so least not today, where they.

E

Need to keep API server and like no I guess.

B

The one way you could do it is if you did a feature gate, but that's like I'm, pretty much like stirring bombs around.

E

To get around the river by some people may actually come to depend on what how event look like right now so like to be able to design with that principle. I'll be so.

B

E

B

E

Delete as into the current events anymore, why.

B

Don't we just table the discussion I think until and take it to the proposal at this point and then I I don't know dan Daniel. If you wanted to chime back in about your prototype, if you had like where, where would it live, what's it doing, is there an issue for it? A topic of discussion.

C

mmm I, don't think there is.

C

But basically, this is my intern project, so I have internal issues. You know everything and I wanted to find something. Just Fremont scale do a pool thing because then the internal become. So that's why you can just revert I. Think I'm, not totally making long-term, because here I have to happen in the same area, doesn't hide long-term well.

B

I think I think if an intern at the throw them in the fire but I, think of an intern wanted to do the hard path of aggregated API server testing. That's a beneficial thing to the community in the long haul to share the API but I'm pretty much like showing Spears at them.

B

E

Then, and ending with these front API every that ID ever kind of assumed the days of auctions.

E

Cells are journey, north available, wonderful of them. It will happen on me or there's no work. You could.

B

Still have you could still like, because it's a t you could still back it int.

E

B

E

Yeah like that, if you like, changing the body, the thesaurus I silicon I'm, doing them with water. Well,.

B

Cassandra supports the ability to do things like watches elasticsearch knowledge, but you know other database systems do now I'm going to do it on the way down which.

E

My goods, and also like updates, which I'm not like up and down big Abidin, also phone update process and I. You can call wait why.

B

Okay, well, it sounds like we can always revisit that topic. I think as things progress, I, don't I don't have time to execute on it at all, recycle.

E

Yeah I use like I, was into the Volga like I'm I, didn't get elected a.

F

E

To like so just like, you know,.

F

Looking for light, shines lighter, but I just for fun some day like yeah.

E

F

Of it, okay, who this is no good? Okay,.

E

Okay, yeah and I guess: that's cool! Look like kind of our timeline.

A

Pretty much I was going to just tough interest to you, mayor I, don't know if anybody here is planning on participating in the website and retrospective on Friday. But one thing I plan on bringing up is the scalability test, basically no attention until like over the tide and release. You have to get a scrambling for the subnet change and we never actually got to run a full-scale good cluster. What I did but look after the releases Todd was my understanding. No.

E

No we're on, like the affairs Bob slide on Monday or Tuesday under we land on the unbelievable, do.

D

A

E

So, are you toward it? Okay,.

A

It seemed to me, like it happened, sort of last minute that in the past, we've had scalability signal a little bit sooner before the release was tough right. Well,.

E

The thing is that it was like pretty it's not that a lot of changed in terms of process. We just like attack a number of issues that appear sorry surface, like actually only in our clusters, which we just there it just don't run continuously Kevin some Kassadin mm, no doctors would find I. Can the court like subnet or whatever or like the event I did answer the notes?

E

First thing he does but emerge just before the release and like well.

F

A

That would be worth praising right, like yeah. No, the right hand was doing there. The other thing I'm, not sure if this would help with it in the future. Is the wood rot proposed that before we cut any alpha or any data, we'll make sure that all those blocking tests are passing I don't actually know if the scalability test is considered a blocking test or not yeah.

E

There's- and we are kind of said, okay I need to read this proposal. Like does axiom is working off on defining which test you want to run continuously and how ID validation process should look like, so that I think they need to signal Phil.

E

Okay, so because I did those tests I'd, like whatever I, run the full blown like all the tests didn't like a week, and let me let you know when to do that project. It should not. Then.

A

It doesn't sound like as a test for paying attention to for Donal go signal right if it will so not only won't scalability.

E

Ties back to just just based on I can have five thousands, no like it was job all right. This kale, just like has this property took.

A

It that's all yeah we're at a time thanks all for showing up I would prepare, as we kind of come back to the table next week, a little more focused on what we want to commit you for one. A so they've got some time to actually put that together before the next date is upon us happy Thursday, everybody.