Cloud Native Computing Foundation EnvoyCon 2019 (San Diego), 27 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Service Mesh in Kubernetes: It’s Not That Easy - Lita Cho & Tom Wanielista, Lyft

Description

Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Service Mesh in Kubernetes: It’s Not That Easy - Lita Cho & Tom Wanielista, Lyft

As Lyft migrated its applications to Kubernetes, assumptions baked into the networking layer were tested. This talk will provide a deep dive of how Lyft used Envoy’s xDS protocol to design their own flexible service mesh and handle new challenges from a multi-cluster architecture such as:
- Routing across multiple Kubernetes clusters
- Handling Deployments
- Rapid scale-in and scale-out
- Service Discovery
- Active/Passive Health Checking
- Readiness in the service mesh

This talk will also go over changes that were made in the Envoy codebase to make this work.

A

Alright, let's get started yeah. This is a last stop before lunch, so we'll try our best to be timely as possible, so my name is Lena Cho and this is Tom. We both work out lifts, I, work on a service mesh and slash networking team and Tom works in the deploys team and we're here to talk to you about our starfish. Most service managed my gracious to Cooper Nettie's and all the specific issues we came around, and hopefully this will.

A

This will give you some learnings so that when you're in your process of your migration, you could take these learnings with you. So we separate the talk into three parts. First is giving you context of our service discovery architecture, legacy versus kubernetes, we'll quickly go through that. So you have the context to understand the issues, and so we broke up the issues into scaling up as well as scaling down on kubernetes, and then we are going to close it all up with future work. That's still left to do so.

A

Yeah our service discovery is fairly simple. How we run envoy as a sidecar and a bunch of virtual machines and there's a con job, that registers ip's into a Python discovery service, that's stored in dynamo and then envoy, poles for that data, every 30 seconds giving the set of IP addresses it needs to start routing traffic to the services it cares about.

A

So this is our hybrid discovery. So again, rather than on belay sidecars polling, we've decided to send the old legacy infrastructure data yeah go into the control plane, so we were migrating over to access control, plane and then also we which is written and go, and we are that control plane also listens to the kubernetes api server. To get the state information of all the pods and IP addresses, and just to note that we don't use ingress controller in our kubernetes cluster, we use each.

A

Will you see an ISO each envoy has a router IP address in each pod. Has a routable IP address and can talk to any parts of a V PC similar to the earlier talk that Thomas gave today.

A

It's basically what we use and then and then the control plane that merges both datasets together and streams that out via xes streaming and to the sidecar on voice, and this is the biggest change here- we're no longer pulling anymore we're streaming data, and this was a conscious choice so that we can resolve the service smash as quickly as possible. So to get the benefits of kubernetes quickly scaling up and quickly scaling down. And then the end of the day of the final state was what we is.

A

We want the control plate to be run in our kubernetes cluster and each krimpet well plan will listen to the get data from the API server and split it. Send that data off to the side car tomboys. It's not clear yet whether we're gonna make each cluster a separate mesh, but at the end of the day, this is kind of what we want. We want the control plane to be, alongside with the pods, sending a discovery data in the kubernetes cluster, so to go into quick details.

A

The control play leverages the kubernetes Informers, which uses the watch api and that's just basically syntactic sugar to handle the watch. Api events, parses them all nicely into nice, go structs and, and we and we use the endpoint objects, so kubernetes endpoint objects, which provides the all the IP addresses for that particular service, and so it provides both old state and new state.

A

And what we do is we just take the new state and over um merge that with any legacy, IP addresses that are available, merge them all together and create es data and then send that out the control, and we, the other key thing that we want to mention, which will be important later in the talk, is that we didn't move to streaming xes api's, and this is so that we could resolve the mesh as quickly as possible.

A

The last piece of information that we need to go over is how we configured our CVS, so we both have active health checking, turned on and ever interval of 15 seconds. We have outlier detection enabled. So if a service or a host is sending a consecutive mine of five five hundreds or five eckesachs, it will get dejected temporarily from the service mesh and we also have panic routing enabled so that is turned on by default.

A

And what that means is that if 50% of the hosts fail health checks, it will start ignoring health checking status and route traffic everywhere.

A

So, let's now dive into the issues, um so we're going to first talk about scale-up, so Ana kubernetes deployment, as most of you know that there is no such thing as like a deployment before in our legacy infrastructure we would hot restart, and so the IP address sets wouldn't change very much. But that's not true.

A

In kubernetes, each deployment is basically a scale-up and that in a scale down so again, we need to we're not propagating way more information in the mesh in time, and so we're gonna say: let's look at each problem separately and we kind of made this story around with invitation to a party so like a pod is coming in. It's like entering a party which is the server smash, so let's dive into scale-up. So it's very important to be polite to yourself and introduce everyone yourself to everyone at the party. So we have during scale-up.

A

Let's say a pod comes up: Cooper nettie, the the cube light or the API server notice is that oh I knew someone just joined the party. So someone joined the service smash and sends an event to tell other people- or in this case your downstream services, that a new pod has entered the mesh, and so it talks to the control plane sends that converts that event into IDs data and sends that out to the downstream services. And then again you have to introduce yourself via health checking.

A

So the down tree services sends out a health check and you notify the ham here. I'm help I'm ready to take traffic.

A

However, so let's go back so we've so, unfortunately, when we scaled up too quickly, we noticed panic. Routing would be enabled, so the detent of panic routing is to prevent thundering herd like his with our legacy. Infrastructure scaling up was quite expensive, so we didn't want all the healthy hosts to get overloaded. So that's the intent with panic routing, but the issue is out during a get appointment or since it's a scaling since a similar to us going up of it.

A

You can see that if more than 50% of 50% of hosts were entered you'll end, it unintentionally enter into panic routing and that's because they enter an unhealthy first all at once, and then you have brief moment panic routing and then it will resolve itself.

A

So working with the open source community was really great. They acknowledge doesn't fix this. It's no create a PR to exclude glued new hosts into health into the load, balancing calculations for panic routing. This works both for panic, routing and priority spillover, and you can- and basically you have to enable it in your CVS file, but this has solved that problem. So now, if you have this, if you have this configuration enabled, it won't include new hosts into the panic routing calculation.

A

So the second issue we ran into was a voice start out. So a lot of applications that lift assume that envoy will be ready to serve traffic and then look it's the infrastructure that was true like when you at deployment happened. You would, what was it you would on boy would still be there. It was an immutable infrastructure, so when it was hot, we started, Amelie will still be running and when you scaled up in her legacy infrastructure, usually on boy was started off.

A

First, then, your application was started, um but that is not true with kubernetes in kubernetes all the site, cars are started up altogether and so, and so it could be the case where your application will be ready to run first and then envoy would and and all we would not be there, and the one solution was to get people to change their code and make an entry, and all that case, but that's not a really great experience for our product engineers and also getting people to change code, will just end up delaying our kubernetes migration even further.

A

So we need to come up with a different approach, so we did two things one is we edited edited what readiness means for the Envoy a sidecar container, so we listen to the ready admin endpoint and what that does is it will wait until all the xes configuration is initialized from your control plane. However, if you have initialization time outset, it will respect that and after a certain time, and if he hasn't initialized during that time out, the readying, the ready employee will send it 200 after that timeout.

A

So that's something to watch out for this. The second thing is that we we ended up working kubernetes so that we can start we can implement ordering and so through that we ended up implementing container ordering in that fork and it's not great um that we're working with the kubernetes community to make that better- and it seems like this is something wandered in industry. So if you want to see what we did, you can look at our fork, and hopefully this gets resolved later later on in the cooper days community.

A

So, in the summary and for a scale up, we had issues with panic, routing but judith of but do do the upstream on boy fix that ended up working out properly and then the other thing is sidecar ordering. So we added ready a more a more representative readiness check for envoy, as well as forking kubernetes, to ensure that envoys started. First, both all the sidecar containers are started first before the application, so great, and so tom is not gonna, go over scale down great.

B

I, don't think I need them.

A

B

Hello, okay, cool yeah, so thank you, Lido we're talking about scale up so I'm gonna talk about scale down and to continue with the party analogy. It's a it's like saying, bye before you leave the party. No one likes it. When you just ghost the party, then people might say like where's Tom and everyone gets confused, just like the service mesh.

B

So let's, let's look at what scale down looks like so you have your pods I've got some happy pods talking to up streams and down streams. Let's say one of the pods gets terminated and it's scaling down for whatever reason kubernetes will see this, send this update to the control plane and then propagate to the downstream. And, of course your health check will also have to fail at some point, looks a lot like the scale-up case.

B

So one thing we noticed when we first shipped this out was that we had this case where we would have kind of like undead hosts in local discovery. So, basically like a pod, we get terminated, but then that IP address would just kind of linger and a couple hosts local discovery, and eventually this. This caused a number of issues, because people just kept getting paged, basically, because the page would basically think that there were some number of hosts that were unhealthy inside the mesh, but they just weren't there period.

B

So something was up, the mesh wasn't converging correctly. We really had no idea what was happening so we dug in so. The interesting thing here is that Ottawa has some level of protection from like a Byzantine failure from the XDS control plane and the what I mean by that is that it kind of respects its health checks quite a bit.

B

So if the control plane were just like barf and tell you, there's no more hosts for something as long as the health checks are still rolling like envoy is gonna, be fine with that and just keep using those hosts.

B

So what end up happening is, is you know, envoy basically has to Ottawa has to see that the health checks have failed and that the hosts are absent from a control, plane, eds response to remove the hose from local discovery.

B

What we found out, though, was that there was a bug here where this had to happen in order so with excess streaming, this kind of made things way more complicated, because, basically, we would kind of send the update and the entry would get stale and we wouldn't send the update again later, because of you know, XTS streaming, it's not periodic the IP address, which is kind of linger. So this is the the table in which boys should respect. As far as like health check status and discovery status, we wanted to be here.

B

We were stuck here and we really needed to get here to delete the the thing, so we needed to fail the health check and be absent. So there's two patches. Thank you Matt for implementing both of these the bottom one I'll talk about first, even though it's the, even though we kind of discovered that bug later, which is why it was later, but the bottom one is handling the health check with you know without any regard to how the ordering works so with that change.

B

Basically envoy keeps track of the state of the last discovery update, and so, if you start failing health checks and in the previous update you got removed, then you know it could clean that that a host out of local discovery, no problem and then another upstream change that we made was allowing the control plane to sort of forcibly remove a host from the mesh. So this essentially allows this extra state.

B

This draining state for you to be able to set in XDS on a host, and so you could propagate that draining state to all the downstream pods and they will know like okay, this pot is about to be deleted. I should stop routing health care, routing requests to it, and once it starts failing health checks or whatever like it's, it's good to clean up.

B

So yeah so now envoys, all good after that patch everything's great it's respecting the the holy table. So even after we fixed that bug, though we started seeing more, we start seeing still some more 503 s, so we were digging into this. We could rely on retries, but we we wanted to do better. We didn't want to just let this happen. So you know, like I, said earlier: we added that draining endpoint.

B

We wanted to take advantage of that and somehow notify the mesh that when a host is going away, you know start draining request, stop sending it request.

B

The reason we saw this 503 issue is because we're using the kubernetes endpoints to sort of be our source of truth as far as like what the eds data is and the way endpoints are handled is that they kind of get deleted. Like the same time, their readiness check fails. It's it's actually concurrent, and in our case the readiness check is the same thing as our Envoy health check. So there could be a case where that endpoint. Well, basically, the health check fails and the endpoint disappears, and then it's just kind of chaos.

B

So here's a high-level diagram of how our control plane looks like like Leo was saying we use an informer, we're using an endpoint Informer and basically the endpoint Informer takes the kubernetes watch API and makes it work kind of like a nice cue. The nice thing about this setup is that on the kubernetes informer side, we're getting these endpoint updates and you can imagine that's for like service a service fee service d. The nice thing about endpoints is that they have the entire state of the world for that service within that cluster.

B

So, basically, the design here is really simple. You just kind of pluck this thing off the queue, and then we shove it into this cache that we have in the control plane and then that cache gets propagated out to all the down screws. Really simple worked well enough, pretty easy! So again, endpoints get removed immediately. Once the readiness check fails, we don't have any time to send this like fancy, draining update we implemented earlier and basically, what ended up happening was.

B

There's a race where the pod gets terminated: it fails health checks because it's gone outright, but this happens before the EDS removal happened. Eds removal occurs and before the act of health check comes in to save the day. So this is the source of the 503 s that we were hitting.

B

So the next logical alternative is to use pods instead of kubernetes endpoints for our control plane, it kind of makes sense. The endpoint controller inside of kubernetes does the same thing and we're kind of replacing endpoints with our solutions, so we might as well do what they're doing so. The nice thing about pods is that they get when they get deleted. They have this termination grace period that you can set and when that's set, what basically happens is the pod.

B

Has this deletion timestamp field that's set at the it's set with the time at which the pod should be deleted. So it's a nice way of knowing that this pot is about to be terminated and we can send a draining update for positive it for pause that we find that have that field set. The annoying thing here is that the pot updates aren't service level like endpoint updates. Are we don't get the whole state of the world?

B

We're gonna, get updates for every single pod and then we're gonna have to maintain a cache as well sounds really annoying.

B

So if you dig a little deeper into the kubernetes client go code, you'll find this neat little thing called an indexer, and the indexer basically allows you to bucket the objects that you're watching and like an arbitrary way. So we have a special label on all the pods within a service, and it's so it's really easy for us to use the indexer to just kind of bucket all those pod updates into the index by service name. So it's kind of like this extra cash, but the cash gets maintained.

B

It's it's sort of like the latest state of the world for for that service or for that set of pods. Rather so, what we end up doing is basically for every pod update. We just kind of pluck out the service name, put that on a queue and then, when we process that queue, we kind of take that service. Name out. Look up that service and the index pull out all the pods generate a cash entry and then put it back into our control, plane, cash and then propagate that to downstream hosts a little more complicated.

B

But we end up getting the same benefit, which is we have the entire state of the world handled by kubernetes, and we don't really have to muck around with our own cash and everything's good, and we have a happy Chihuahua.

B

So, in summary, scale down the the two things we kind of had to tackle here was tackling the out of order, bugs notably around discovery in health check data and then also getting our control plane to monitor pods rather than endpoints, because that seems to give us time to send the draining update and it allows us to avoid yeah exactly just sending the draining update, basically so everything's good. We got all these cool cool check marks.

B

So in summary, everybody likes talking about paradigm shifts. So I will talk about the paradigm shift as well, where the changes essentially are moving from no level changes like Lita was saying earlier in our VM infrastructure, the VMS would kind of stay where they were and we were kind of update things in place, but we're sort of moving the those levels of changes with kubernetes into the network, topology or the network topography, and so that's to take advantage of kubernetes quick scale up and scale down and again few things needed tweaking to get there.

B

We have to patch kubernetes, which was not ideal, but it's what we did and and yeah and as far as future work goes, Thomas from psyllium had a very interesting talk about running on voice, a daemon set. We run on boy as a side car in every pod, and this is expensive for compute reasons. It also, you know, makes the mesh that makes mesh scaling more difficult. If you have even more on voice to deal with seems interesting seems complicated, it's probably something that will dig into later.

B

The other thing that we're looking into is that we have this like one giant mesh or essentially, two giant meshes one for staging one for production, and it's it's hard to propagate that information in time across to all the all the downstream hosts when things are changing, especially when things are rapidly changing in kubernetes.

B

So we're looking into a couple strategies, one of which is cutting up that mesh, maybe cutting them up by AWS availability zone or by cluster and another is batching the updates, so that in case there is some big scaling up or scaling down event. We can just batch those updates and send them send one update. Instead of sending you know a thousand or however many we have to and really interested in some of the envoy OSS changes with Federation and supporting multiple control planes.

B

So thank you, and now we have an invitation to a party for you and we, hopefully you say, hi and bye to us appropriately. Please join us at the lift happy hour on Tuesday November 19th at a bar called thorn barrio Logan, which also sounds like auto-generated password.

B

Please RSVP'd on the website, or you could just show up at the door and you'll be fine yeah. Thank you.