Cloud Native Computing Foundation ServiceMeshCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Service Mesh: A Hole in the Pocket? - Venil Noronha & John Murray, Stripe

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Service Mesh: A Hole in the Pocket?- Venil Noronha & John Murray, Stripe

Service meshes provides a wide range of capabilities including flexible configuration, high fidelity observability, and is transparent to user applications. However, this power, despite it's (often) open-source nature, does not come cheap. In the age of public cloud, vendors typically charge users on several axes of usage. When using a feature like traffic shaping, if not careful, cross-availability-zone traffic can quickly increase spend. High fidelity metrics are useful to operators, but default service mesh configurations can easily overwhelm systems, and users too. Suboptimal defaults coupled with user-local dashboards leads to high cardinality metrics further increasing costs of metrics storage solutions. Yet another area of spend is access logging. While being extremely important to diagnose runtime issues, access logs can quickly become a $$$ drain. The default policy of logging each request is not the best configuration for most users, and organizations need to craft these policies based on their unique environments. In this talk, we will talk about costs we've encountered when using a service mesh at scale at Stripe. We will also discuss some strategies we have in place for reducing spend, and other techniques to improve the overall experience with service meshes.

A

Thank you for attending our talk today, uh service mesh, a hole in the pocket a little bit about your presenters today. My name is john murray. I am a engineer on the service networking team at strength. I'm building our internal service mesh. I am an occasional envoy contributor when they need arises and outside of work. I am a sequel plus enthusiast.

B

And I'm vanilla narona I work on the service mesh platform at stripe. I've been a contributor in the istio and envoy communities and I really enjoy distributed systems.

A

Okay, so let's jump into the process of service mesh first is simplified application network radio features like load, balancing circuit, breaking retries and the service measures. You know about all these teachers in a way that's really easy to take advantage of for application owners right. It may be simple, specifying a few headers on the request and this works across different languages and frameworks.

A

Next is traffic patterns, um so you have like things like deployments: traffic splitting traffic shaping fault injection.

A

These are similar to kind of the application networking features before, but they're more organizational wide right, so, whether it's for reliability or for like advanced deployment scenarios, these are kind of features you're going to want, as your company scales security, which, for most people, is going to look something like mutual and tls uh or sorry, mtls uh and role based access controls um and then lastly, observability right metrics access logs, distributed tracing- and you know, as your organization grows, you're going to want a lot of these features.

A

A lot of these controls you're going to want uniform interfaces for all these things uh and the service mesh kind of solves this, especially given that, as company grows, we typically see more languages, more frameworks, kind of being used, whether it's through evolving development practices internally or maybe it's through something like acquisition um and so having a service mesh, unifies all of these interfaces into kind of one access point both for control and development.

A

Okay. So what are the costs of running a service mesh at a high level? uh There's explicit costs, so this is the cost that we typically think about up front when we're doing a research and we're deciding on whether we want to adopt a service mesh or not. uh There's the hidden costs right. The costs that you're, typically not going to think about before adopting you're, usually going to find this after you've adopted, implemented and kind of ran with the service mesh for a while.

A

uh There's integration cost right, which is uh what does it cost to kind of hook? My service mesh up to two other third-party services, we'll see what those are in a little bit developer costs. um You know it's it's not necessarily free for developers to to build on top of the service mesh. So what does it take to provide that level of education um and uh productivity that you want to deliver to your developers for them to be effective?

A

uh And, lastly, sport costs right service, mesh, uh isn't gonna run itself, um so so what does it cost to maintain operate a service mesh?

A

Okay, let's jump into the explicit costs.

A

uh First is cpu um right uh and uh under the here, I'm just configured quota right because it's sometimes too easy to maybe provide too much cpu and you're just kind of burning money or try too little, maybe you're, impacting kind of your latency uh overhead uh same kind of goes for memory. Where you can, you can allocate too much and you're just kind of burning money allocate too little and you may be impacting the performance of your applications.

A

uh Latency uh misconfigured concurrency, so you know different ways you can configure uh your service mesh and how much resources it takes up uh can also impact kind of that latency um yeah. So these are. These are the calls. These are things we think about right, like how how much money is it gonna? Is it gonna cost to actually run this new code on all of my machines um and what is the the latency I'm going to expect to add to all of my requests that are all the requests that are fluent in service mesh?

A

Okay. So now, let's look at the hidden costs. uh First, up network bandwidth usage, uh particularly thinking about control, plane traffic.

A

um So the the configuration data you ship from your control plane to your envoys, uh depending on your fleet, size may be fairly large, maybe in the order of megabytes, if you're operating a significantly large fleet, maybe gigabytes and depending on your topology, you may be shipping this data around to thousands, tens of thousands, maybe 100 thousands of nodes right, and so that's all um kind of network bandwidth usage that you have to kind of add into your costs now next is I o costs, so the the service mesh makes a lot of things, implicit and maybe previously more explicit.

A

So an example here might be availability zones, whereas before maybe your code accounted for crossing those availability zone regions with a service mesh, you have to be careful on how you configure it because you may be crossing those boundaries without knowing it and paying those additional costs.

A

There's also like feature specific costs right. Health checks, especially active health checking, is, is sending more data across the wire across your network to check all of the health of the upstreams.

A

This can kind of turn into a star pattern, sometimes with service meshes. So the uh you have to be careful about how much traffic you're generating health checks um and other features like hedging and retries these. These are reliability features. These are probably features we want, but we need to understand that they're not necessarily free and we are going to be generating more network costs by using them.

A

Okay. So next up is integration costs, uh so we can think first with like metrics right, the service mesh uh is is generates a ton of metrics. um It generates metrics that are uniform across a bunch of types of applications, so they're highly useful for debugging your system understanding your system. um However, sometimes they can generate a rather insane amount of metrics, uh probably more than you need um and uh you're gonna have to think about uh the cost associated with storing those um and the vendor cost right.

A

So, if you're uploading to the cloud, those kind of your vendor costs, if you're, even if you're running your own internal metric system like a prometheus store, you're going to think about all the prometheus clusters, you're going to have to run to support the metrics coming from your service, mesh uh log storage uh is the same way. um Service mesh can produce uniform access logging, which can be super useful for for debugging and troubleshooting heterogeneous systems. um But again, like you know, that's it can get very costly very quickly um and you can be generating.

A

You know terabytes or more of data every day every hour. uh So it's definitely something to consider and in the same vein as log storage is distributed, tracing right, uh you know you you're going to be generating a lot of different spans. uh So you need to think up front if that's a feature, you want to use how much you're going to allocate for storage costs, okay, so developer costs.

A

uh This mainly goes down to education, um so you're going to need to kind of train your developers on like how to debug and troubleshoot certain errors that may arise from the service, mesh or use of the service mesh and then kind of teach them code patterns to use and avoid right. So this is kind of really understanding uh the interfaces of the surface mesh and how to work with it.

A

um The the mesh is not always as transparent as it may be, marketed to be, or, as we may hope it to be uh so education is important uh and, lastly, just kind of managing resource quotas right, especially if you're running in, like a like a kubernetes type system with with uh maybe like an envoy sidecar as an example, um then you're gonna, your developers are gonna need to know uh how to appropriately set quotas for the service mesh um and what the trade-off is there between uh cost and maybe performance great all right, so support costs.

A

uh First, up is kind of maintenance right. um This may be something like cve patches. uh Any things you have to do. You have to stay secure, so you need to spend the time to to patch your surface smash with a lot of all the fixes uh you may have other patches as well. They're, just kind of like bug fix patches that you may find uh service mesh typically represent a lot of different traffic patterns, and not all of those patterns have equal coverage from your upstream, like uh provider of that open source software.

A

uh So you may need to find your fixing stuff yourself, sometimes uh api compatibility and upgrades. So uh looking at the envoy ecosystem as an example, um the xds, the migration from b2 to d3 involved a lot of work and potentially down the road. A v3 and b4 migration in that configuration language will also involve a fair bit of work, which kind of scales as your fleet size grows, operations, uh data plane, upgrades right if you're deploying a mesh and a sidecar pattern.

A

This may just be a lot of nodes to update a lot of time spent doing those upgrades control, plane upgrades they may fall into the same boat. uh You may be using a central control point, um in which case your upgrades are faster, but they're also much riskier more dangerous, so there's just additional time taken to make sure you get that right and things like sort rotations like these things. All these things just kind of take time.

A

uh Troubleshooting is the next big one. uh We talked about developer education and how that's important for our users to understand how to troubleshoot errors and work with service mesh, but their service measure locations can just be a lot. You know they're built very generally to serve a lot of different use cases, and- and so um you know, when there's a critical service having issues it can be really hard to be to tell them like go. Read this giant body of documentation and, like you'll, be able to figure out your own problem.

A

Sometimes you just have to jump in uh get things fixed, get things up and running, so your team operating service smash will probably spend a fair amount of their time, troubleshooting and lastly, ownership of client libraries again in developer education. We talked about patterns and practices uh when working with a service mesh, you may decide that it's easier to take those patterns and practices and implement them inside of a client library that you distribute to your users. But again, this is you know another form or another another way of cost.

A

uh For that particular issue that you should consider uh so that wraps us up for costs, I'm going to hand it off to vanilla now to talk about strategies for controlling cost.

B

Thank you, john. Let's now have a look at some strategies for controlling costs. I will first give you a overview of different strategies, and then I will dive into each one.

B

The first strategy is to understand, defaults service meshes and proxies come with default configurations which can affect the proxy's behavior of routing requests or the amount of data generated by these service meshes. Therefore, it becomes important to understand these defaults in order to control costs.

B

The next strategy is to sample access logs, depending on the amount of traffic flowing through a system. Your service mesh can generate a large number of access logs.

B

This can quickly increase your storage costs and having a good sampling strategy for your access logs can help you cut down these costs.

B

Similar to that as metrics, because service mesh proxies can generate a large number of stats, it can quickly overwhelm your metric systems. Therefore, it becomes important to simplify matrix in order to reduce costs.

B

The final strategy here is to perform easy. Our routing ac stands for availability zones, cloud vendor vendors typically have policies for networking and io, and when your requests cross easy borders, it's going to cost you. It becomes then important to identify services that are routing across acs and also ways to minimize cross easy traffic.

B

Let's now have a look at some details onward, for example, has default http 2 configuration, for example. You can configure max concurrent streams or you can leave it at the default.

B

You can also configure on voice concurrency, which basically is the number of worker threads. That envoy must use for request. Processing onward also comes with default, retry and circuit breaker configurations which affects the way requests are routed to your system by default onward generates a lot of stats, so it becomes important for you to figure out which of these are meaningful and which ones to discard.

B

If you, for example, misconfigure onwise, concurrency or http 2 configuration, it can degrade performance, thereby increasing your costs.

B

If you leave circuit, breaker or retract configurations to default, it can affect the way your requests are routed through your system in a failure scenario and that can increase your network usage, thereby increasing costs and, as I said, metrics by default, can overwhelm your metric system. So understanding the default behavior of your service meshes or proxies is important to cut down costs associated with metrics.

B

You typically see a lot of 2x6 responses in your system. These are not very interesting, so logging them can definitely overwhelm your system, but not provide you good insights.

B

It therefore becomes important to identify what is your use case for access box. In some cases you may want to use access logs for incident remediation, but in other cases you may want to use this for long term learning, for example, to visualize a service graph of all the components in your system.

B

You can also think of sampling or filtering access logs, depending on your need. In the case of stripe, we completely discard all 2x responses and we also discard all responses associated with health checks. By that I mean we do not log any of these kind of responses.

B

Service mesh proxies generate a large number of stats and they can quickly overwhelm your metric storage system.

B

That basically means that you will be spending a lot of money. Just on your metric store storage system.

B

These metrics can also increase. You increase cognitive load because these come from disparate sources, so it becomes important to simplify observability for service owners to easily understand the state of the system using metrics.

B

You can think of simplifying matrix by either filtering out irrelevant metrics or you can also think of combining metrics from different sources in order to provide a unified view, so that it is easier to understand them and at the same time you do not store all of these metrics, which can increase your costs at stripe. If we do not control the way metrics are generated, it has the capacity to quickly consume a majority of our metrics budget.

B

Finally, let's have a look at how asia, where routing can increase your spend cloud vendors charge for traffic that crosses easy borders, so you need to identify which services really route across azs. This could also be just the control plane communication itself.

B

Once you identify which services route across easies, it becomes then easier to identify whether that is really required, or you can cut that down.

B

This is one of the benefits of using a service mesh that is now you can control services and their routing behaviors, so that they can prefer local services versus ones that are routing that are running remotely.

B

You need to take care, though, that if there is a local outage, your service is still up and running, that is by preferring a remote service over a failing local service, and for that you may want to implement health checks, or you may want to employ outlier detection.

B

Let's now have a look at some open problems. The first one is troubleshooting. When you deploy your application on a service mesh, it becomes a little difficult to figure out where problems actually exist.

B

Whether the problem is with the application itself or whether it is with the network or whether it is with the service mesh, is a little hard to tell. Therefore, you may want to spend some efforts on building tooling to aid troubleshooting.

B

One thing that comes to mind here is to hand over the ownership of client libraries to the networking team. By doing that, the networking team can instrument these client libraries to aid troubleshooting, but at the same time, it becomes difficult to figure out where to draw the line between this team owning network interfaces versus business abstractions.

B

Another important aspect is developer education. What can we do to educate developers about service mesh behavior? Here we are talking about application developers. Can service mesh? Is the 100 transparent to service owners, or do we need to build higher level abstractions or write huge documentation in order to educate developers.

B

Finally, another important point is maintenance. How can we simplify upgrades envoy releases are typically done every three months, and there are cv patches that come in between deploying these to a large scale system can be time consuming.

B

Also when there is a api upgrade within on void, it can become difficult to migrate between one api to another.

B

How can we minimize this friction? Also hard restarts may not be available across different onward versions. So how do we ensure that upgrades are smooth? These kind of questions are still open.

B

In summary, we had a look at the promises of a service mesh costs of running a service mesh. Some strategies for controlling costs, some open problems, and now a big question remains that is: is it worth it?

B

We do think that service meshes can solve interesting challenges for you, but you need to be very intentional about what features you require from a service mesh. You need to understand what features to enable what are its defaults? What sort of logs and metrics can these features generate and how much of your budget can it consume?

B

With that, we would like to end the talk. Thank you. Stripe is hiring so feel free to reach out to us. If you are interested, and now we are open for questions.