Cloud Native Computing Foundation ServiceMeshCon EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tune Your Service Mesh - Mohammad Reza Saleh Sedghpour, Umeå University

Description

Tune Your Service Mesh - Mohammad Reza Saleh Sedghpour, Umeå University

Service meshes improve developer productivity by factoring out cloud native patterns such as retry mechanism, circuit breaking, etc. into a unified network control plane. Modern cloud platform adopters realized how tough the tuning of these patterns can be in a highly interdependent and dynamic microservice architecture and how improper configuration of such patterns can reduce throughput and/or increase the response time. This talk will discuss the pitfalls of configuring circuit breaking and retry mechanisms in a multi-tier application using Istio. It will help you configure your environment systematically to enhance the performance of your application from an end-to-end perspective.

A

I'm mohammed is a alternative picture student and uh uh I'm uh working in uh autonomous, distributed system labs in umea university in sweden and in these days, I'm mostly working on autonomic microservice management and especially in traffic management policies.

A

So, let's begin, as you know, the performance of microservice microservices that we are working on these days depends on various factors such as complexity of each service that we have and the incoming workload and many other things that we are familiar with.

A

Therefore, the management in such dynamic environment could be really complex, but the good news is that service measures are there to help us to handle this complexity in the fly.

A

uh But uh let's look at an example, one of the uh amazing complexity. One of the amazing capabilities of service measures uh is traffic management policies uh which enables us to have secondary care, retry patterns and without the need to implement anything in our development process.

A

So, uh let's see, let's see them in practice.

A

I want to ask you how many of you are using service mesh in production, nice great and uh how many of you are using uh traffic management policies such as second parking and reply mechanism in production.

A

There's a difference that people are not really working with these uh things that and now we are going to discuss that uh and uh yes that was it okay. Let me uh uh share my shocking experiments. uh Experience uh that I've had during my research uh there's a possibility to ruin the user experience by uh misconfiguring uh these two patterns.

A

I mean the retrieve setting and uh circuit breaker pattern, and the worst case uh is that uh they can even uh cause uh outage outages some outages in your services, uh but they can also help us to maintain our slas.

A

uh I would say that uh these patterns are to some extent like a doubleh sword, so we have to be careful about them, but let's explore them in practice. I've done some some experiments with these patterns and I'm going to share them right now for the experiments uh I use the online boutique, uh which is a simple e-commerce, app that you might be also familiar with it's open source implemented by google, and I use the cart endpoint and.

A

While it invokes a chain of three marker services, as you can see here, for enter recommendation and product catalog and in total, we need for a single and complete uh response.

A

There is a need to have six requests during all of these three services.

A

So I deployed this uh online project to a governance cluster with this with istio installed, and I've manually identified three key parameters to highlight the worst case: scenarios for the for both circuit breaker and reply setting for circuit breaker. I use the http to max request, which is the maximum number of requests to a backend service uh or based on my experience, and my understanding is like a cue size.

A

But let's for simplicity, let's call that a second breaker configuration for this stack and uh for retry setting. I use attempts, which is the number of requires to be allowed for a given request and also prototype mode, uh which is the time of per attempt for a given request, including the initial call and then retro is that we have, but based on my understanding, is like an interval.

A

So we can call them interval attempt and a second vector configuration.

A

Okay, now we reach to the uh main experiments that I've made for my talk.

A

uh Concert consider that we've done everything and we deployed the app and we are going to study. uh Where should we enforce our circuit breaker.

A

In the first scenario, I uh enforced the second breaker in the first year in the frontend service and uh with 20 as the size of the queue, and then I generated 1.2 times of the maximum load that service can handle. You know with a response time less than 100 milliseconds, but let's see uh what will happen to the services in the first year, as you can see here in the first year, uh there's about 200 requests per second successful requests, and, and also there are about 60-ish, 60s or 70.

A

Second workout request and the same throughput is also in the second tier, but we had quadruple uh uh throughput for third tier because it depends on the architecture of our service online magic. I mean.

A

With keeping these values with 200, I mean in mind, we are going to enforce on other tiers, to see what will happen in practice and also the response. We have to see the response times uh in in this scenario. In this case, we are going to enforce the same circuit, breaker configuration on the second tier, and you can see that there are about 260 requests per second successful request per second.

A

That inputs got higher and also the same to put for a second here.

A

But there are some circuit broken requests in second here, which are seems like a fade request from a client's outside clients, uh clients, perspective and also the same thing happens for the third tier and then for the next scenario, we're going to enforce the second breaker on the third tier, and we can see that we have about 220 requests per successful request and about 50 or 60 uh freight requests, and the same thing happens for circuit breaker, requests which are in the chart here, but uh the our clients see them as a faint request.

A

Now, let's enforce this second marker on all tiers.

A

If, as you can see here in the first year, we can see around 200 210 request, successful requests per second and some circuit breaker request is somehow similar to the situation that we enforce our circuit breaker in the first year.

A

But it's it was not all. uh Let's look at the response time uh here is the cdf chart of for all experiments that I've uh done the response time for uh the successful request or or known as carry to put, are highlighted with a circle marker and for all requests, including the uh failed ones, are highlighted with a triangle marker, as you can see, when reinforced the circuit breaker on first tier or on all tiers.

A

You can see that here the response time is below 100 milliseconds, mostly, and uh if we enforce the second breaker on uh third year or second year, we can see that the response time are higher or for second year is. It goes up to 500 milliseconds, it's huge, so that was uh one one of the experiments.

A

Let me check yes, uh but let's see what happens uh when the incoming traffic is changing, which is the actual environment that we have, uh but uh let's just skip the short presentation, because it's take it takes time and uh uh and focus on what what lesson we can learn from this case. For this case, the circuit breakers is enforced on the first tier, with different generated traffic load, as you can see there.

A

Eighty percent one hundred percent and 120 percent, uh and uh but what we can learn is that uh having higher incoming traffic can lead to uh more successful requests and also circuit record circuit, broken request. Also having circuit breaker in the first year could help us to have fast, faster failure so that we could have better user experience in a way and uh our system can maintain the unacceptable response time.

A

uh But one other case is to investigate the impact of circuit breaking values. I mean the size of the queue that we had for this case, uh reinforced the circuit breaker with different values on first year with an overloading traffic again.

A

But what we can learn from this case is that there's a trade-off between request rate request rate and response time.

A

I mean if we we have higher circuit, breaker values, then the request rate increase increases, while there will be higher response time and vice versa, or we could say. Second, second breaker somehow protects latency at the expense of availability.

A

And the final takeaway from this case is that the best value for circuit breaker for this app and this setup that we had is 20. for the queue size uh now it's time uh to see the what happens. If we have reply setting as well, in this case, the second breaker with different values and one retry, attempts are enforced on the third tier with an overloading traffic again and we didn't set any retry intervals for this case to see the impact of this combination.

A

uh Again, we can learn that uh the same observation as previous case that there's a trade-off between request rate and response time uh when configuring, the second breaker thing and when we configure a lower circuit breaker or there are more fade requests in the downstream services. There will be more requests to be retried, but in general the retry setting could increase the response time when typing circuit breaker or there is any failure in our in our in between our services and.

A

uh Let's deep dive into the uh retry setting and see uh what happens if we change the number of attempts for our read price uh and uh but to somehow highlight the worst case scenario. We configured the really sensitive circuit breaker, we limited the size of the queue to one to have more circuit, breaker requests, and what we can learn from this case is that the higher retry items can increase the throughput slightly, but with higher response times.

A

And now, let's look at the intervals, what if we have different intervals between each retry? In this case, uh we can learn that uh uh uh high retry intervals uh does not really affect both response time and request rate, but having no intervals uh could cause a required storm in between our services.

A

uh But but we have to keep in mind. We are using http to max requests in istio and we don't have any uh replication set or other things, just one single service.

A

Okay, let's see what happens if there are uh some spikes in the incoming traffic with different intervals, which is expectable in in a live environment.

A

What we can learn uh from this case is that a high return interval again does not affect the response time and request rate, but too short returning, two short retry intervals might cause a slightly higher number of uh circuit breaker requests, especially after spikes that we had. On the other hand, if the loader spikes are larger, we could see uh more circuit, broken requests with higher response time or somehow we could call them short retry storms after loaded spikes,.

A

Okay and as the final case, let's take a look at a case where we just change the enforcement point of these two patterns that we have replace. I think and circuit breaker.

A

When we enforce the optimal circuit breaker on first year or on all tiers different enforcement point of view, price, I think uh plays no role because our clients, uh our traffic generator, didn't support the retry setting. So uh it didn't recharge, so it doesn't have any uh effect. But when we enforce the second breaker on second tier, uh we can see that there are. There will be higher number of wait, requests and higher response time.

A

If we enforce the retry setting on all tiers, uh we can see that the the the higher number of failure requests. On the other hand, if we enforce the circuit breaker on third tier, uh reinforcing their retry setting on first tier or or on all tiers, the highest number of affair requires and highest response time is observed.

A

So, let's summarize uh what we learned and how should we configure uh retry setting and circuit breaker.

A

How should we deal with such dynamic environments, so in general we can say that finding the optimal configuration can be really tricky, because uh our uh the performance of our services are changing dynamically, with the incoming traffic with the cicd pipelines that we have. So we cannot say that this is the best configuration for all services. We have to tune every time, so maybe a controller for that.

A

Also. Based on these findings, uh we can conclude that if we want to have low response time and highest possible throughput, it seems to be beneficial to enforce a more uh sensitive circuit breaker in the earlier layers, maybe in the first layer as well, so that this could enable fast failure and better user experience.

A

About retry, uh it is not really suggested to enforce high number of required attempts, while it might cause withdrawal storm if we have uh some sort of bad uh misconfigured circuit maker, or we didn't configure any recline turbos.

A

So and about refined, turbos orbiter said uh high retry intervals does not really play a huge role under throughput, but the short retro interval, as uh we've seen, could somehow increase the require storm possibilities and those uh situations uh and one of the benefits of retry setting is that if we don't have a vlt on circuit breaker in our system, retry setting, I mean both neutral interval and retry. Attempts could somehow increase uh the throughput during overload, but with a little bit higher response time.

A

And finally, the most important point about these two settings is that they could be really good choices if we have, if we, if we have uh transient failures or transient overloads. If we have a larger for larger previous overloads, then we have to think about other scenarios like such as uh scaling or other things. So that was all, and that was my research somehow thank you.

B

Does anyone have any questions for muhammad? We are actually ahead of schedule, so it will be really great if you have any questions.

B

Oh right, um so we have one gentleman have a question: actually it's easier. So when you have a question, walk up to the middle there's a microphone there, so you can ask your question there. Thank you.

C

Hello, have you done any research into tuning retry jitter, not just the interval.

A

uh No, I didn't do that, but I'm trying to do an english in my next phases. Thank you. Thank.

C

B

Okay, any other questions.

A

B

Okay, I'm not seeing any. Let me know if I miss anybody. If not. Thank you again. Thank you.