Home
Contribute Contact Us Browse all meetings
Home Contribute Contact Us Browse all meetings
Cloud Native Computing Foundation / ServiceMeshCon EU 2021 / 14 May 2021
Cloud Native Computing Foundation / ServiceMeshCon EU 2021 / 14 May 2021
Previous Meeting Next Meeting
⏯
Add meeting Rate page Subscribe
youtube image
►
From YouTube: Lightning Talk: The Hardest Part of Operating a Service Mesh: Envoy Proxy - Christian Posta, Solo.io

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: The Hardest Part of Operating a Service Mesh: Envoy Proxy - Christian Posta, Solo.io

Operating a service mesh in production has some pragmatic complexities with some contributed by Envoy’s underlying powerful feature set. Requests flow over the data plane proxies which means this is an important piece of technology to understand how to operationalize, debug, tune, and observe. In this talk, we share our experience at Solo.io supporting customers on Envoy-based technology both at the edge and in a mesh and share lessons learned such as: * understand which telemetry signals to watch for config sync issues, CPU, and memory pressure * enabling access logging and enhancing it with dynamic metadata * debugging connectivity issues with Envoy logs, metrics * tuning for cloud environments with keep-alive settings Attendees will leave with a better confidence of running Envoy-based service meshes such as Istio, Consul, Kuma, and OpenServiceMesh in production.

A

Hello and thanks for coming to my talk at servicemishcon eu 2021, I'm going to be talking about service mesh, and this is a topic that I've been very interested in for a long time now. um I got involved in the ongoing seo communities back before you see.

A

I was even ga and I've been working with customers and and users of istio and other service meshes actually across the world at large and small organizations, helping them operate and productionize their service mesh and and continue to modernize their application infrastructure and envoy proxy is the center of this talk today, because a lot of service meshes use envoy under the covers and for good reason envoy is an open source project.

A

It's nc, plus very fast, very feature rich and was built with a dynamic api to drive its configuration, so no flat files and hot reloading and all this stuff, but on voip, proxy or any service proxy under the covers in a service. Mesh kit is on the request path and can be kind of complicated envoy can be run as a sidecar, as I've mentioned in a service mesh pattern where the proxy lives with the application instance in kubernetes. This would be with a pod in a vmworld. This would be envoy.

A

Living on a vm envoy can also live at the edge of a boundary and do reverse proxying type type, load, balancing and routing and so forth and envoy, when it's in the request path, especially for developers who are not familiar with envoy, might see it as a black box, and this is not specific to envoy. This is any any service mesh data plane proxy.

A

So what happens when things start to slow down or don't behave, the way you're expecting?

A

If there are issues, how do you actually troubleshoot it, and that is one of the most important and sometimes the hardest part of operating a service mesh which is understanding the data plane.

A

So in this talk, we're going to take a look at a few tips and tricks that we have learned and I've learned over the years, helping folks operationalize envoy-based technologies like service mesh.

A

So the first thing you should know is that envoy is not a black box. Envoy is a white box. You can see into it and you can uh glean a lot of information about what's happening inside the proxy on the request path over time.

A

So, first of all, there's an admin interface, an http admin interface that you can uh query for things like certificate information, things like what what upstream clusters that are routable and what endpoints make up those clusters.

A

Things like changing the log levels, which will become very important as you're, trying to debug things as the requests are going through envoy and really important features like being able to tap the request and response chains, as well as being able to do profiling and memory dumps. So there's a lot of really important functionality. If you're going to operate envoy, if you're going to run a service mesh, you should be familiar with getting to this admin interface.

A

When you need it and understanding how to query it and how to get information out of it, the next thing you should know about uh operating envoy is in the search part of the admin interface or in the stats part of the admin interface. There are indicators that will give you a heads up when things might start to go wrong. So one of those things that we've found the hard way is keeping an eye on when certificates are about to expire.

A

Now, tools like a service mesh site like istio, might have a way for uh for doing automated rotation of certificates, but sometimes that doesn't happen properly and checking these certain stats, whether you have automation in place or not to determine you know if you're getting closer and closer to assert expiring and it not being rotated when you expect it extremely important.

A

Another is what happens when the envoy proxy starts to come under severe load. What is the behavior that you expect? We don't want the envoy proxy users to just lock up and behave. You know uh in a state that we can't understand is unpredictable, so we can set things like the the overload uh manager in envoy and and observe it and watch it. How are what is the pressure? What is the memory or cpu pressure that envoys uh at right now and what should it do when it gets to certain thresholds?

A

Should it stop doing keep alive on certain connections? If you uh reach a certain threshold, should it stop accepting requests, should it start shedding requests and so envoy? Has this this feature called overload manager that allows you to ahead of time specify what happens when the proxy starts to become under pressure? So you can start to reason about understand. What's uh what's happening, here's a here's, a snippet or example of what it might look like that.

A

You know you specify we're going to monitor the the heap space and if it gets to 95 or crosses that high water mark, then we're going to disable keep alive, and so what that means? If, if there are connections along the connections, open, there's not really much happening on them, but we have keep alive there so that it keeps those connections open. We're gonna, we're gonna, stop to keep alive, and those connections should eventually get cleaned up.

A

If we hit 99 or across that high water mark, then we will stop accepting new requests and so on right. So you can check the documents, but there's a really powerful feature and when running onboard, another one is being able to log out to standard dollar to a logging file when requests are coming into the system and log metadata about each of the requests.

A

Things like certificates, things like x, forwarded for headers uh things like request and response details of the of the message and headers and so on all extremely extremely useful for um debugging and troubleshooting when things start to go wrong in an envoy based environment.

A

Another incredibly important piece to this puzzle is not just the access logs, which we talked about so request comes in. We can log metadata about that, but it's also the proxy itself enabling so the so envoy has extremely uh detailed logging levels or or rather different modules inside of envoy, that you can enable logging levels for and some of the common ones around connection connection handling.

A

What the filter chains are doing, how routing happens, how our back policies are applied and so forth? Those can all be enabled at very fine grain levels, and things are not connecting for tls issues for some reason go check the envoy logs. That will give you a very good hint, if not exactly why things are not working and then one of the last slides is, you know tuning for cloud deployments, especially in a public cloud.

A

One one of the most common one that we've been running into and have been bit by for the last several years really is, uh is how the load balancers behave in a public cloud. So if you end up running envoy proxy on a vm and you end up running a data or a control plane somewhere else, you know connecting those things to an amazon load balancer, you could see very unpredictable results so being able to tune things like keep alive and session and stream uh keep alive. These things are are very important.

A

um Here's a little snippet for how you might do that for an upstream cluster, so maybe you're connecting to the control plane for configuration, updates or you're connecting to the next service part of the control plane to determine authentication or authorization policies.

A

When a request comes in configuring, these things is extremely extremely important and you know, like I said these are all these are all tips and tricks that hopefully help you we've learned them the hardware hard way uh we have the the scars to show that um and if you're interested in running envoy based technology, either at the edge as in an api gateway or a service mesh like istio and uh you're. Looking for a way to make that successful and simplify doing that, please reach out to us at solo.il. That's exactly what we work on.

A

That's what we specialize in it's, where our core competencies and we'd be happy to to help. So with that, thank you and uh go enjoy the rest of the talks in servicemeshcon.