Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro + Deep Dive: SIG Scalability - Marcel Zięba & Wojciech Tyczyński, Google

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Intro + Deep Dive: SIG Scalability - Marcel Zięba & Wojciech Tyczyński, Google

This session will focus on the different efforts that SIG Scalability is involved in: defining what scalability means for Kubernetes, driving performance improvements, maintaining infrastructure for scalability testing, guarding Kubernetes against performance regressions. Time for Q&A will be reserved at the end of the session to understand how the SIG can better engage with the community as well as to allow the audience to provide the input about the roadmap.

A

Okay, um hello, everyone uh thanks for joining. um I know it's friday, so I appreciate you are here. um My name is wojtek taczynski. I work in google, I'm the tl of sixth scalability and together with martial gimba, who is one of the chairs of six scalability. We are going to to present, like the sixth scalability update today, um okay, so, let's start so uh what we are actually doing, what the scalability is actually doing.

A

So there are basically five main aspects of things we are doing, which kind of correspond to sub projects, but uh um in terms of like six scalability that doesn't really own any non-test related code like sub projects. This um sub projects are somewhat somewhat fluid here, so um so yeah. So the first thing that we are doing is actually defining. What does scalability really mean for kubernetes um it?

A

It both means like defining defining what does it actually mean, but also like where we are actually heading like the core principle for any scalability related efforts is to um focus on actual real life scenarios, and real life needs, like optimization, for the sake of optimization, doesn't really make sense, because it's usually with some exceptions, but usually it's like complicates the system. So we should be only focusing on things that are um really needed for someone.

A

um The next thing that is like strictly related to that is like once we have those goals like we need to actually execute towards those, so um so actually driving and ensuring that, like the actual improvements that are needed to to to get to those goals are, are um are something that we are working on.

A

um This doesn't really mean in many cases that we are doing the improvements, um if, if those are fitting into um individual six, like I don't know, signature or sig api and missionary or whatever, we are trying to um to work with those six and ensure that those will be doing that, but we are often um also contributing to those themselves and for any crosstic improvements. Then there were a bunch of those like over the past years. um We are, we are usually coordinating those across those six too. um The next thing is.

A

Ensuring or checking where we actually are like, we know where we would like to be based on the first item, but um we don't really often know where exactly we are now um so monitoring and measuring, where exactly we are is, is pretty critical to to to understand how far we are from our goals and is there anything? Is there really any work that we we have to do in this area?

A

um The next item, which is strictly related to that is, is ensuring that we are not regressing. So once we reach certain level of scalability, we need to ensure that that we actually stay at that level. It's it's pretty pretty easy with scalability to regress, um because, like every every new feature is or every new change in the system is often, um if not well thought through it, it's often possible to easily regress the system.

A

And finally, we need to ensure that scalability is is not the job for like a very small group of of people like sitting in the corner, but it's actually a job for everyone like similarly to reliability or security or whatever like when you are designing your designing and implementing your your feature. You need to be thinking about like scalability on your own, like we like it can't be. We can't ensure scalability um if the overall community won't be thinking about that. So we actually made a very, very long journey and very successful journey.

A

I would say, since the since um the early beginning of kubernetes, when there were pretty much like two or three of us like thinking about scalability now, um scalability is actually an inherent part for pretty much every single cap which is like I I I hope that you all know what cap is this? It's, like our design, dock, more or less um people are thinking about scalability, but um not everyone knows exactly how to do it. They know they should be thinking about it.

A

So this this item is actually about ensuring that, when they reach out to us like we have like best practices, we um we can recommend what they should be doing in that situation and so on and so on. um Okay, um one more thing probably worth mentioning is that, like we, you shouldn't be confusing it or it's often confused like stick out.

A

Six scalability is often confused or mixed with sick out of scaling like auto scanning is about scaling either your size of the cluster like add by adding nodes or removing nodes or um scaling your number of pods or or pots in general, either horizontally or vertically.

A

um Scalability is about like the amount of stuff that you that you can do or amount of of, for example, like how big your cluster is, or how many, um how big services you can handle in a single cluster and so on, and so on. um Okay, so um what is really scalability in terms of kubernetes? So, as I mentioned like you, should all we should always be focused on on actual user needs.

A

So when you ask users if they want scalability or they, if they want this cluster to skate, the answer is obvious, like they are they're all saying yes of course, but if you know, if you ask them like what does it really mean, um the answer is often like. I don't know, or it's even more. It's often also, I don't care because they don't want to like understand all the details of the system. um They want us to handle that, so we need to define it our on our own.

A

So like the first approximation that we did like in early 2015, um is we approximated scalability with like size of the cluster, so number of nodes? Basically, so when I first started looking into that, like in february 2015 or something like that, like the 25 note, cluster was basically blowing up um and over time we are basically improving that, like um starting from like support, 400 notes in 1.0, um reaching like 5 000 notes um in the steps like in in 1.6 release, and we didn't really go higher than that since then.

A

um So now you are now. You are probably thinking like if we did anything like in the last like five years, and yes, we did. um We did quite a lot because, like um scalability is like actually much more than than like the size of the cluster or number of nodes, um basically scalability is like a multi-dimensional um multi-dimensional or scalability in kubernetes.

A

It's a multi-dimensional problem where, like things like number of services, number of pods number of end points in a service number of persistent volumes, and so on, like the all, they are actually matter for scalability.

A

um So we introduced this concept of like scalability envelope, which is basically saying that if your cluster within a single with within each of those dimensions, actually fitting into that scalability envelope, that means that, like your cluster has like scales or your cluster will be happy basically.

A

um So what does it really mean that the like your cluster skates or? Is it happy so we are building or we are defining that like based on like two main concepts? um It's a license is a low, so sli is like silver service level indicator as the lowest service level objective. You can conceptually think about them. As like, sli being a metric and the slo being like a metric and with threshold, as it's actually much more subtle but like conceptually, it's it's basically, um it's basically that so.

A

We are saying that, like the cluster is happy, if all those scalability slows are basically satisfied, so more or less like all the metrics are within that the threshold that we that we defined. So um we don't have much time, and we don't have like um enough time to talk like in very details about like slis and slows.

A

So um if you are interested, I had a talk purely in that topic purely about the topic in like barcelona, you've gone like so you can probably find find that in in youtube, if you want so just very briefly like we have like six main slos that we are currently measuring um slice and slos, they have like sub varia.

A

Some of them have some sub variants, but like um um conceptually, we have six of them like the first two are with us like since the um very beginning, um the the the latter four um were added over time.

A

So, as you can see like, we don't have like a super large coverage. um There are different pieces of system that aren't really covered. Like I mean some of them are covered implicitly like, for example, like scheduling latency.

A

um It's actually part of pot. Startup time or pot startup latency, so um there is there's more cover to like um implicitly than expo. Then then, then, from what looks it's from the one. From from how it looks like from the first glance, but um we are still, we still have a bunch of like holsters. So it's one of the things that we want to invest for quite some time and we we never like have time for that. So um if you are interested like it's, it's definitely one of their ideas that we would.

A

um We would benefit from your help. um So let's take maybe one one example and look into a little bit more detail.

A

um I'm not going to to to read it, and I I don't even expect you to read it like the the main point is that um even though, like the the conceptually the sli should be, um are we are trying to make them very simple and that, like high level, so basically for mutating api calls, we can say that, like the, um what we want to achieve is um that, like the mutate, like the 99 percent of mutating api, calls is finishing one in one second, and this is more or less what is written in the slide.

A

But we need to be super precise in the slos to ensure that the way we understand it and the way we measure it is like exactly how you, as a user, also understand it's. It's like we had a bunch of cases or I've heard about other cases, also from outside kubernetes, where um the the fact that, like the the ones who defined the slo um understood it differently than the actual users was, was causing um a lot of friction. So we need to be super precise here, um oops, yeah, okay.

A

So, um basically, once once we have the like, once we defined like all those slos users know what to expect where what to expect from the system or how the system will be behaving if they are within the scalability envelope, but they also like or maybe primarily in many cases, they want to know like what the scalability and below or how, how large it is or how, how far they can go in in the dimension um they are actually interested in. So um computing is precisely it's like super hard or maybe even impossible.

A

So we are not trying even to do that. We are fortunately able to like approximate it pretty well um by by by providing like a certain thresholds in many dimensions in all of the dimensions um it it's. If you, if you take a combination of those, um it's it's not actually the whole whole envelope, it's possible to go much farther in in certain dimensions if you go lower in other dimensions, but we want to provide something that is, that is easily consumable for everyone and for the most sophisticated kind of most sophisticated users.

A

We can work for them and explain them better. Now like, if they can actually go farther with that, um okay, and with that, I'm I'm passing the microphone to marcel we'll be talking about other stuff.

B

Thank you so, basically, right now we know what scalability means. What kind of sls do we care about? But the question is like: how do we measure it?

B

How we ensure that this, as these slos are actually satisfied in kubernetes, so we will go through scalability testing infrastructure, what kind of tools we use and basically how we ensure that kubernetes is scalable, so our main tool that we use every day is cluster loader two you can think of it as bring your own cloud, bring your own layout, basically kind of similar to like regular deployments that you that you do for kubernetes and you can think of it as you specify kind of states in which you want to have your cluster in, for example, let's say that you want to make sure that you can run 200 000 pods in your cluster, then what you do is you specify this state as, for example, you create 1000 deployments with 200 pods each, and then you can specify even more states like okay.

B

Once I have those 200 000 thoughts, let's say I want to delete half of them and when you have those states what you can do is you can specify how you transition between those states, because, obviously, if you create 1000 deployments with 200 000 pods in total, then most likely something will break.

B

But what you can do is basically say: okay, I want to create one deployment per 10 seconds, or maybe you want to specify that um you want to create all those 1000 deployments within 10 minutes or one hour, so you can specify all of those transitions between states and then on top of that, once you do it. What you can do is measure measure those slows that we mentioned. So you specify that, for example, you care about api, latency or pod startup latency.

B

You care that the pod startup latency will be within 10 seconds and so on, and except for that, we have a bunch of other extra features that allow us to monitor, observe and test it. Basically, you can find all of that on our documentation for cluster loader too, and if you ever want to scale test. Kubernetes, I strongly recommend to use this tool, but of course, as you may imagine, running, cluster node is pretty expensive and in open source.

B

What we do is actually we run 5 000 nodes every day to test for scalability regressions, but this doesn't give us like really good observability, because, let's imagine that you know there is one day and there are 50 prs merge into kubernetes and we see some regression.

B

This would be quite hard to kind of like see which kind of pr's are actually breaking the kubernetes, but fortunately we have cubemark and cubemark is simulation of the cluster instead of running 5000 nodes. What you can do is just run like 80 nodes and until on those nodes, you can schedule parts. For example, what we call whole nodes and hollow nodes is something that simulates regular node, but it doesn't actually run the pods, so you can think of it as okay, regular kublet just takes the pod and it's running the container but whole node.

B

Just reports back to api server, I'm running the container, but it's not actually running the container. So whole node consists of actually three parts.

B

One is holocubat and we also have hollow cube proxy, because kubrox is actually one of the parts of kubernetes that is putting quite a lot of pressure on the api, server and kubernetes master, and except for that, we have also hollow node problem detector.

B

So, with this kind of setup, what you can do is with 600 600 cpus. You can test or simulate 5000 node cluster um yeah, but then you might ask okay. So how? How do you actually run those 5000 hollow nodes? It's it's not easy right. So what we do is we actually we actually create second cluster, so we have one kubernetes cluster that is responsible for uh for just running those hollow nodes and then those whole nodes connect to the actual kubernetes master that we are scale testing with cluster loader, two yeah, except for that.

B

What we do is here. You can see our great tool perf dash, as you can see, it's quite outdated, but it's like one of the most useful tools that we have uh in six callability, and here you can see example that okay.

B

This is 5000 um ci test that we constantly run for for kubernetes, and we measure here, pod, startup latency and you can see like 99th, percentile, 50, percentile and so on and based on that, uh we can detect some possible regressions, because slo says that okay, the startup latency 99 percentile needs to be below 5 seconds, but um we are still unhappy when you know the 99 percentile was three seconds and then it bumped to four seconds, for example.

B

So we we see it in the perth dash and based on that we can still debug it and see it.

B

um Then, of course we have graphana. We are collecting like bunch of metrics. I actually recommend to to check it out and those dashboards are in our our repository and you can use it for your own cluster. If you want- and there is like bunch of like really cool graphs- that allow you to to check like api server. Latency but scheduling and and all of that stuff.

B

Then we also have profiling. So when you are running cluster loader two, what happens is that uh it's not only like gathering those uh prometheus metrics, but also it's getting the profiling from the api server, for example, and based on this profiling. uh What we usually do is we go through it and see what kind of parts of api server is actually consuming most of the cpu or memory and based on that we can plan and iterate on improvements to api server.

B

So now, okay, I talked a little bit about our infrastructure. What kind of tools we use for for scalability testing, and now we can go back to scalability test what kind of tests we actually do run and what you can see in our test grid. Basically, so we have periodic tests, ci tests and we split them into two categories. One is release blocking and one is non-release blocking, so release blocking tests are the ones that you know there is new release of kubernetes and we see that okay, the scalability doesn't look great.

B

Then we say: okay, we just need to stop the release of kubernetes and debug it. What happened basically and we have performance tests with 100 nodes performance tests with 5000 nodes and correctness tests that make sure that regular features in kubernetes work at scale as well, except for that we have non-release blocking which are kind of like more informative for us. So cubemark is one of these examples that I talked about, but except for that we have storage, gohank benchmarking different type of, but one of my favorites is actually going so with going.

B

We actually saw like multiple times that you know going changed the compiler changed and it totally broke kubernetes.

B

So we have one dedicated test just for just for testing compiler of the going, which is basically running fixed version of kubernetes, and but we are just changing the calling compiler and if you are a contributor, then probably you saw our pre-submit.

B

We have uh 100 notes per submit test that it's running for each pr, that um someone is trying to merge to kubernetes master, and this is like early warning that something might be broken basically and it protects basically kubernetes from like super super big regressions.

B

So yeah now, let's get to protecting scalability of kubernetes. So this is our that's great. You can check it out. It's uh it's in, uh like you know like basically the test screen that all the tests are there and you can find like six scalability test grid um and we test also release branches old ones and to make sure that you know. If there are some cherry peaks to 122 123, then they are also not breaking old versions of kubernetes.

B

As whitehead mentioned before, scalability is very sensitive and what happens is that there are so many things that can break scalability of kubernetes, to name a few. We saw, as I mentioned before, going that compiler changes can break, but also operating system controllers and api machinery, meaning like api server schedule, scheduler at cd and cubot, basically everywhere.

B

So what we do is we try to try out those issues, debug them and once we know that okay, the fault is because of the change in scheduler or going. What we do is we reach out to those six and try to to help them fix those issues, and sometimes we just fix them by ourselves.

B

um So I will give you a few examples of like really cool regressions that we recently debugged. So one was spot startup latency, and the idea was that, okay, since, like 120 we've been heavily investing in priority and fairness and along the way there was one regression that significantly increased the pot startup latency. So we were the ones that were debugging and the root cause was actually quite simple. So um we started supporting, watches in priority and fairness and the number of routines that one watch was requiring was actually doubled.

B

So, instead of like having one thousand one hundred fifty thousand of routines, we had three hundred thousand guardians and, at this point, scalability just degraded for for pot startup latency there's also like api called latency regression, which was also connected to priority and fairness. This actually was.

B

This actually basically increased the api call latency for all the possible resources, and so we were also debugging it and fixing. There is pretty cool debugging uh in this issue that you can. You can find on github so, except for protecting from scalability regressions. We are also driving scalability improvements, and recently we were helping with the migration of going to 118, and then we also help with implementing efficient watch resumption or immutable secrets, like secrets, are actually one of those things in cluster that can put quite a lot of pressure on api servers.

B

So if you are deploying um your workloads and um and let's say that you are using secrets, then maybe you should consider using immutable secrets to just make your kubernetes api, reliable, more reliable and except for that, we also work heavily on priority and fairness, which further increases the reliability of kubernetes.

B

So if you want to get involved, there are bunch of links. You can attend our public meetings, they are on thursday, 17 30. and yeah join our slack channel. If you have any questions or mailing lists, you can, you can always reach out to us and we are happy to help.

B

If you are developing new features, then we can help you with reviewing it or if you have any issues, then we can try to help you with debugging those issues and if you want to get involved, there are some issues that are marked as help wanted and basically getting started. And so, if you are interested in our tooling, then you can help us with developing tooling. If you are interested in regressions, you can help us with debugging regressions.

B

So thank you and now it's time for q, a.

C

So nowadays, there are sometimes more crts and contrasts in a cluster than than workload or anything else. So how do you? What do you think about this and how do you consider it.

A

Yeah, so um I think there are different aspects of that like well. Well-Designed controllers with cr like based on crds, are not really a problem.

A

um I mean crds are a little bit more expensive than like built-in apis, because because of not using protobufs, for example, and using json and so on, but like that's not a core of the problem like we, it's fine like the biggest problems that we see are the like, node but note oriented controllers, so that, like the demo, sets more or less that are like watching or even worse, listing in some cases like a bunch of state from many nodes. So so that is really what is causing the problem.

A

So I think that our main problem is not the fact that we have crts and controllers. It's the fact that, like um still, there are people that are designing the controllers in like in an inefficient way. I would say um I think we we would like to get to like improve like the efficiency of protobufs in general, like the the discussions about supporting protobufs for crds were happening since I don't know four years or something. It's just like big thing with not high enough priority at this point.

C

D

Hi uh my question is: um where are uh the trade-offs or focuses in terms of improving uh scalability um or or where are the red lines that you would not like to cross?

D

uh More concretely, for example, have you considered rewriting some of the components in another language or or switching out a very important, important component or deprecating a feature that impacts scalability a lot, but a lot of people rely on that.

A

Yes, so um good question, so um I think the first example that that you, that you made, for example, rewriting components something that we really would like to avoid. I don't think we want to do that. I mean especially in a different language. I think we want to like consistency, um it's fairly important like for the project as a whole like for us as a community and so on, like we, we don't want um to diverge too much like between components. So I think it's it's.

A

It's probably one of those lines that we don't want to cross, um redesigning individual components.

A

I would treat it as a last resort. I, but I wouldn't exclude it like, for example, networking sig network is actually considering or they are, they are thinking, and there there's even kept that it's like um not yet approved, but like active on like how to redesign cube proxy to make it a little bit more efficient.

A

So like those kind of things, is something that we um definitely consider but like it should go into like individual six like the um the requirements should be coming from us, but, like um it shouldn't be us doing the work, it should be that sick driving the work um to ride that controller and- uh and us may like, like in coordination with us and like us, helping with that but like driven by them more or less. I would say um sorry I forgot the last part of your question.

A

I think there was something more at the end. The.

D

Last part was about deprecating features. Oh.

A

Yeah tom, I think.

A

The goal is to actually uncover the regressions uh or uncover the unscalable things like as early as possible, so like in general, like purely from scalability like wearing my scalability hat like yes, I would like there are some things that I would like to deprecate, and I would like to get rid um wearing more. My like production, readiness, hud and like, like all those like higher level things. um We really don't want to do that, like we really don't want to break users.

A

So um what we are trying to do is we are trying to introduce a different way of doing stuff or the more efficient way of doing stuff. For example, like um podante affinity, is one of those features like of one of those like scheduler features that doesn't really scale well and it's causing a lot of troubles for scheduler.

A

We introduced like the feature that is called how it's called bot topology spread. If I remember correctly, that is kind of doing a very similar job or like for majority of cases. um So we are trying to steer people towards like more scalable regressions by not by disallowing them to run, to use the old ones, but rather about like giving them a carrot to to use those new ones or you are a better one or whatever you call them there. There are exceptions. I think we recently deprecated like the selfling field.

A

um That was part of like, like every object used to have like a self wing. That was basically like a self link right um and we we actually deprecated it. It's no longer set anymore, um but that was done based on like significant research and and the outcome that no one is really using it for anything useful. That and of like all the cases where that we are aware and many different repos can easily be replaced with already existing stuff and that are not based on stealthing.

A

So um that's the only example that I remember in the last like two three years where we did that, um and we generally would really like to avoid it as a project.

E

Okay, oh sorry, sorry I I have two questions. um Firstly, I think uh way at the start you mentioned you have some documented design principles for other maintainers, so uh or maybe I misunderstood.

E

So if, if as a uh as a component maintainer in in kubernetes, I I want to go and read the the you know the cliff notes. uh What what should I be thinking about um when making design changes to my component? Do you do you have a general principles.

A

We are it a little bit like it's. There are some pieces, some small pieces here and there written, but I think um one of the things which is we are missing like this go to page that you can go and like read, those are seven things that you should first think about it and if you are interested, then in more detail, let's shot, but um we are actually missing that really.

B

Yeah, but also like, if you are uh developing uh some component, then I would recommend like just adding it to to our load test. So it will be just covered by our regular ci tests, which can also be helpful for you, if it's possible, of course, because it's not not always possible.

E

I maintain cloud components so um yeah normally not but um okay, um so my other one was was kind of related to the previous question, so I was kind of interested. If you could, you described diagnosing a pod, latency issue and the root cause was, I think you said, a change had doubled the number of go routines.

B

Yeah yeah yeah.

E

So um presumably somebody had a reason for doubling the number of go routines. This.

B

Was just like you know, programming back. Basically that was introduced. Okay, you know and uh still like the slo was fine right, so we didn't see it at first, but using the perv dash where we can see like the changes over the time, we were able to spot that you know 99th percentile of pots, turtle latency increase from like three seconds to four seconds and based on that we were able to kind of like you know, see that okay, it was within this time that the start of latency increased.

B

Then we went through a few pr's and we picked this one that it's most likely this one and then we work with wojtek to fix it.

A

Yes, maybe to tweak the answer. I wouldn't because I'm the one responsible for the progression.

A

I wouldn't say it was a buck. It was basically a thing like how to like when you are like implementing a feature or designing a feature you want to get to like to production as soon as possible. So um there are like there are some shortcuts that you are sometimes kind of like taking and it it was like kind of like unaware shortcut that, like we like, the tests were passing. So we we didn't really pay that attention, um but but yeah it's like it was basically um don't over optimize.

A

If it's not really not really needed, it appeared that it's actually needed. So so we like fixed that but yeah.

E

Okay, so yeah you didn't, you didn't, have to ask somebody yourself in this case to uh to remove a feature um uh or or redesign something.

A

Yeah, unfortunately, it was relatively small change, so it's it wasn't like really like conceptually like we are paying like. We are trying to pay as much attention as possible to the design itself, and once the design looks scalable, we should be pretty convinced that, like the implementation is possible to do in scalable way and like even if the first attempt is not maybe scalable, then you you can re-implement it somehow or pieces of that um and um keep the same semantics, which is, which is the car right off of it.

F

um Hey so you mentioned about the 5k nodes number and basically the edge scenarios are pushing that number quite to the edge and I'm thinking about what. What is your approach about? How to tackling the increasement of eventually nodes? That's running in higher latency environments,.

A

um I think we've never really focused on like the high latency environments. To be honest um in terms of like increasing the number of nodes like uh gke, which like occurs like google, managed kubernetes, we already support like 15 000 nodes. So it's and it's built on top of kubernetes. We need to do a bunch of, like other things around, to make make that really work, and that was like a lot of work, but um the building blocks from kubernetes in open source kubernetes are there. So um it's not that it's impossible to get there.

A

um If we didn't see enough use cases like or people in in open source community asking for more um so we didn't. We didn't try to do that in open source, because there was like a bunch of other things to do in open source to to make that work, but um but it's certainly possible if, if we will start hearing those requests um but yeah, as I mentioned, that we've never really were looking into like um edge cases or like the high latency or isolated things, or things like that.

A

So um there is some work happening in that area, not in the context of scalability, but I'm not super familiar with that. To be honest,.

F

Okay, thank you.

A

Okay, any any last questions. Oh there is your head.

G

Yeah, um what do you think the biggest constraints on scalability are going forward with kubernetes like if you wanted to do 50 000 nodes? What are the biggest challenges you think you'd face.

A

I think there are many different steps, many different things depending on like where exactly you would like to to go. So I can imagine like going to 50 000 notes with not much effort if you, for example, run just like simple batch pot per node workloads without using any sophisticated and working storage, or anything like that. That shouldn't be that hard, um especially if, like a low term cluster and so on, so it really depends like.

A

What are the? What are the dimensions that you would like to you would like to to stress more like usually, networking stack is something that is like most stressing the control plane because there are um most of other things are basically um are basically, okay, maybe taking a step back.

A

So we should split the the problems on like control, plane, components which, which there are like very few of them, or I mean very few per cluster, not very few components, but very few per class, like very few instances of a single component per cluster and those that are running on nodes um and those that are running on nodes are usually like contributing like the highest number of like load on the control plane um and from those like the networking ones are usually the most um the most stressing.

A

So so this is. This is something that we definitely need to. um We need to come up with something if we really need to go, go visibly higher than that um in terms of individual components. It's usually boils down to um to throughput, or things like that, like on the individual components which which is kind of solvable at the per component level, so I would say being able to horizontally scale.

A

Server is something that is that is fairly critical and that we may want to invest a little bit into that more. um The storage layers of the lcd itself is is potential bottlenecks and then, like the components themselves, will or can drive like improvements. If we really need to go higher, which is in dimension related to that component, which is usually will be throughput or the time to process, all all the objects of a given type or something.

A

G

Thank you very much for coming.