Cloud Native Computing Foundation EnvoyCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Trials and Success of Adopting Envoy at Tinder - Yuki Sawa & Cooper Jackson, Tinder

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Trials and Success of Adopting Envoy at Tinder - Yuki Sawa & Cooper Jackson, Tinder

Tinder adopted an Envoy based service mesh for their Kubernetes based infrastructure a few years ago. Our implementation is custom to Tinder's architecture but the lessons we picked up along the way are universal. Whether it's how we store and represent our routing configuration, building out a robust monitoring pipeline for all our Envoy metrics, utilizing Envoy's global rate limiting capabilities to protect our infrastructure, meshing our database and microservice requests, retry and timeout do’s and don’ts, and more, there is plenty of knowledge to share. We will explore our bespoke features like deadline propagation, multi-cluster Envoy metric monitoring, and how we implemented our own xDS control plane. From the early struggles of our service mesh migration, to cautionary tales and best practices, we're excited to walk through the process of building out an Envoy topology that powers Tinder's infrastructure worldwide.

A

Hi everyone thanks for coming to our talk today, so we'll be talking about the trials and success of adopting envoy tinder, so I'm yuki, I'm engineering manager at tinder. I'm part of the service mesh migration team, and this is cooper.

B

Hi so I'm cooper and I'm a senior site, reliability, engineer, tinder and uh we've been working together for around three years.

A

Yeah um so yeah. This is just kind of the overview. What we're going to talk about today just get started talking about why envoy some of the migration hurdles or some of the human costs with the migration cooper's got some nice before and after mesh charts for you. So some you know grafana metrics of what our services look like after the mesh migration we'll talk about how we did some of the network tuning, like you, know, nitty gritty, retries and timeout tuning some of the observability.

A

So how we, you know, incorporated uh envoy into prometheus stack some of the the war stories, so you know things we figured out only after working with envoy for a pretty long time, we'll talk about how we do our envoy configuration um a bit about our control plane and then, lastly, just talk about the future work.

A

So you know today we're fully on envoy and kubernetes took about like one year for the full migration started. Around early 2019 ended early 2020 and right now we have 1.6 million rps meshed in our infrastructure. So that's like internal and external requests over 200, 000 containers, 20, 000, pods and 500 services, um and this is just kind of an overview of how the you know the infra is laid out.

A

So you have you know the user on one end using the tinder app and then requests from the tinder app go to one of three kubernetes clusters and each cluster is shorted to an az. So you know you know the cluster at the top cluster. A is entirely in the 1a available availability zone in aws, so it's wholly self-contained and all the requests get processed within that cluster.

A

So you know that gives us a lot of savings in terms of cross az costs as well as latency, and as you can see, you know what you've amplified pretty much every part of our stack at the ingress layer, we're using envoy within the kubernetes cluster we're using envoy as a sidecar in all the pods, and you know we have. We have like some elbs albs engine x laying around, but the core network logic is implemented within envoy.

A

So you know why envoy: it's got a lot of great features, so we didn't, you know, use all these features right in the beginning, but we we do today use, I think, almost all these, you know multiple algorithms for load balancing. We try circuit breakers timeouts various database filters which we use for observability.

A

It gives us you know metrics for free and then some tracing so cooper's. Just now going to talk about where we came from.

B

Yeah so before envoy, we use really conventional kubernetes networking, so we did flannel for our potter pod network and we used uh regular kubernetes services. You know cluster ip node port with a couple of headless around very few elbs, but just mostly a standard cluster rp.

B

This worked really well out of the box and helped us scale, but we really ran into performance issues, so our clusters were pretty large at the time around, like 4000 nodes per cluster and flannel started using really really high cpu usage. We also saw stale tcp connection tunnels with ipvs, and we also had hanging connections which caused undue load.

B

We ended up with a lot of uneven load on target pods and target services which really really pushed us to go to envoy next. So we started migration envoy in 2019, so we had a lot of services still in the backup. You know services, clutch rp and we had all new ones using mesh, and so we ran a hybrid model for quite a while.

B

uh This worked relatively well. There are some teething issues with that which we can get into later, but once we started going on envoy, there was no turning back. It was so much better. Our observability increased massively our traffic, the pods was significantly better. We had much easier sinking of like load, balancing types of you know, rod robin maglev, etc, and even latency just magically got better.

A

Yeah so some of the migration hurdles we went through right, so we started really with zero envoy knowledge across the entire org. You know, you know we just knew kind of what was out there in the docs and the blogs.

A

You know which is kind of sparse, because it's new technology right, but we did eventually, you know, end up learning it pretty well over the last two years um and we migrated over 300 services over a year just started with, like one person working on it, but had like six to ten people working on it by the end. Full-Time was definitely a team effort. um You know, there's multiple people involved uh people working on integrating it into kubernetes.

A

uh There's a team working on the observability, the prometheus stack, a team working on the you know: configuration automation, integrating it into the ci cd stack migration of the services itself.

A

You know people working on the control plane and then you know people working on the network tuning right and, of course, the service owners were, you know, initially hesitant with the migration. uh Definitely a time sink, but as people saw the benefit right, they just started to volunteer because it was such a you know. So many things you got for free and then uh you know the process just became really streamlined. Pretty low effort didn't require someone with high envelope.

A

Expertise to the babysit and all their services today are by default, have service mesh enabled unless there's a very good reason. I don't think we have a single one that doesn't have mesh disabled now um and then, of course, critical high traffic services were pretty hard to migrate.

A

uh I think we, you know we spent like a couple weeks, just tuning timeouts and retries for certain services, so there's a lot of babysitting on those like high rps ones, and then currently we have 500 microservices in the mesh 20 000 pods migrated and every request at tinder now goes through a service mesh.

B

All right, let's talk uh some nice little graphs here, okay, so before and after mesh, these are some old grass. I pulled from 2019, so bear with me if some aren't perfectly clear, they're kind of old, so our latency was so much better after the cutover. I added a nice little green line, so you can tell boring before and after if you can't see that closely, but you can see that, like our p99 latency massively took a downturn afterwards.

B

Much of this originally was just due to better pod distribution.

B

So before, as I said, with ipvs and flannel, we would have one pod be super hot and super skewing the p99 up, but after envoy really brought the low distribution uh to be way more, even cpu usage again uh following up the last thing about load distribution became significantly better uh for the cold service. This was only after we were partially meshed. So I would say about this point: uh we're about fifty percent of the callers were meshed and you can see the per pod.

B

Cpu distribution become much tighter and uh much more coherent as the time went on, especially after that first major cut, which I think was around 30 percent of its calls.

B

So how did we get there and what do we do to get better, so we did uh in addition to just envoy, we started adding timeouts and retries, so we didn't not have this before, but it wasn't very consistent. So we had modules in our kubernetes cluster, whose own code handled retries and timeouts per application.

B

We had to remove that because you cannot have per application retries and timeouts as well as envoy. You get a huge, uncontrollable, potentially bad situation. If you do that, if you have improper timeouts, you may be either pre-maturing cancellation calls or even potentially, worse you're doing like a retry storm or one retry can cost two more tries and so on and so forth. So we had to do a coordinated effort between back end code and operations to clean up timeouts and make sure we disabled retries, uh due to some even legacy code.

B

Some code owners didn't even know they had it. So it really was a you know: drill down through a whole bunch of code to figure out what exactly was there and what needed to be disabled, but once we fully got everyone off of that, moving to envoy, uh timeouts and retries was cynically better. It was easy, centralized place. We could tweak it significantly better than having it in code. In addition to timeout retries. We also do egress lists via envoy which locks it, which locks our mesh network down securely cool.

B

So, in addition to the load balancing graph, I showed you before. We also did a lot of tuning and timeouts and retries. We didn't get to quite to do hedging retries, but just cutting off the top of the p99s and making them retry at a faster rate, really helped p99. Overall, from what I mean is, let's say we know that service is going to get back in 500 milliseconds and we have errant pods. uh You know maybe doing two seconds.

B

We just cut it at 500, milliseconds or 600 milliseconds and make him retry and that retry will be faster than just. You know waiting for that bad bod to come back and it really sliced down the top uh p99.

B

So one of the things that I love about match, besides just the resiliency and I'm kind of an observability nerd, is that it gave us so many metrics. So on the old old, uh I can't read that sign.

B

It's pretty dark in here.

B

Okay, thank you. So, on the old side of our network, we had some metrics given us by modules about http that again was not consistent or unified across everything between node between go between java between teams. Even we did not have centralized metrics of http.

B

Let's say we had about 30 percent coverage in our kubernetes cluster, which is a massive monitoring gap envoy immediately I mean as soon as we onboard even onboard even a couple into envoy. We got great metrics. We had http codes per route, we had latency, we had retries, we had adapter consistency, metrics we had everything we ever want even per pod.

B

It was great like there was no no longer wondering. Oh, the service is getting some 503s and finding out that, oh, it's one pod, we know exactly which part it is now over a standardized method.

B

We use grafana heavily we're a very prometheus, focused organization and we pipe all these metrics into our prometheus and thanos setup. We do really cool things like we have canary dashboards and canary, deploys that use prometrics using onboard metrics. They can say instantly that pod that just went out is worse than the deploy that was already running.

B

This one is very hard to see, but if you look at the top right, we can see that the canary is doing like 30 better than the last one. So we have all these really great mesh metrics that allow us to do good, deploys and good observability.

B

Another fun thing we do with mesh metrics and standard case metrics like c advisor and coop state, is like standardized dashboards dashboards per team. What I mean is, there's no longer does a team have to immediately just make their own dashboard to track their metrics.

B

We go through an automated process that if you make a new module, you get pagerduty alerts, you get slack alerts, you get everything you want and it's fully based off mesh, metrics and uh case metrics. They can have their own if they want to and we encourage it. But everyone gets these base dashboards.

B

We also do uh filters for dynamodb, redis and elasticsearch, which is the tweak http filter. So we get really good insights and metrics on those services as well being called for mesh services yep, okay. So what didn't go well, as I mentioned earlier, the timeouts and retries needed to be locked in and when they weren't it was kind of rough.

B

uh Since we do a microservice organization, we do many calls to go from. You know, microsoft's a to b to c to d, and if your timeouts aren't scheduled correctly, and especially if you don't have cancellation propagation in your code, you could potentially have like 13 tries for every try. You start at the beginning.

B

So you need to make sure that you always go down that each chain is can be encapsulated by the one before it. Otherwise, you can do way more retries than you could ever think possible.

B

So we did filters, we know this. What else did we learn on this? So we learned that if you best make sure that envoy is the first to start and the last to stay alive, uh since we are doing egress and ingress through all of them. If envoy is not up, when you start and not not still up, when everything goes down, you will drop connections. You will drop requests uh if envoy goes down first, when it both get the sig term.

B

When pod gets terminated, their in-flight requests on the uh on the module will get dropped.

B

Not only do we find out that we need to do that, but we also found out that on the cluster in general, it was nice to catch the sig term and let's say if a pod gets terminated for at least 10 to 15 seconds before you even pass it through no matter, if you're, using like old school ipvs or ip table services with coop, proxy or you're, using the new eds or xds api servers, there isn't they're, not always instantaneous, so just stay alive, a little bit longer to catch up with a couple that are kind of slow to update.

B

So we originally didn't do this and it caused a whole bunch of issues with deploys. So what I'm talking about is on kubernetes. If you do a you know, pod terminator pod roll uh once it gets sent the termination call, they send a sync term down to all the all the containers in the pod.

B

If you start shutting down immediately, assuming even that your code is handling, grace will shut down properly. uh You're, probably gonna die in about a second or so, even if you do a pretty good, diligent job.

B

uh We learned that, even after that, second, that there were still extraneous callers that took a little bit to update from various sources so that dead pod would still be getting calls uh to combat that. As I said, we did a 10 to 15. Second, wait on that sig term. So, even though all those pods were determined to be, you know terminating, they were still well enough to handle traffic for another 10 15 seconds to really just mop up the last few requests that brought down our 500 class codes by almost 100 during deploys.

B

So we have a truly really easily fun deploy.

A

B

A

I'm just going to talk about global rate, limiting since it's, I think, a fairly unique envoy feature since it requires it's like an external service as well as, like a redis back end, a little different from the other envoy features, and this was also like a pretty big win for a pretty big win for us, um because previously you know various back-end teams, they would write their own rate, limiting logic right with their own server and their own.

A

You know database to track the rate limiting state, but with envoy we're able to just you know consolidate that all into the envoy network layer, so we have global rate limiting right, not per server or like per service.

A

We have granular configuration, we can rate limit on one or more headers and then, of course, we gave we got monitoring, alerting and visibility into the rate limiting, which was, you know, often an afterthought when people you know hand rolled their own rate limiting services, so that was great. So today at tinder it prevents system failure. You know stops fraud and a lot of cost savings from not having to process those fraudulent requests and it polices about 200 000 rps at tinder.

A

You can base it on. You know: user id ip, any kind of headers, so very useful. I mean we have two rate limiting clusters. We have an internal one for internal requests and external for external requests, each with their own redis, backend and rate limit service, and then the redis request, or even proxy through envoy.

A

um So this is like a cool little chart. I was able to you know make with the observability um that envelope gives us right. So we we can see where all the rate limiting is happening. Like you can see in the middle east, there's some pretty big dots, I think in europe.

A

A lot of the attackers come from europe because I think that the ip laws are a little looser than the u.s, and you can also see some pretty big dots around where the aws data centers like virginia and the west coast are so envoy configuration. um You know I'm just going to talk a little about that, because it can be pretty complex.

A

Long right, you have like you, could have like a thousand line, long gamma and we needed a way to just surface the minimum number of knobs to the service owners that weren't really knowledgeable about envoy, but they still wanted to like timeout to retries themselves right, so we created an in-house dsl, so the service owners just write a short yaml snippet, and then it gets translated into an envoy config and how it basically works is right.

A

There's like a ginger template that gets converted into an envoy, config yaml gets baked into a config map, that's rolled into our kubernetes cluster, and you know we're able to just empower the owners and they can do self-service smash tuning this way.

A

So the dsl supports retries timeouts rate limits, tracing outlier detection, ip restriction, direct response, data filters, and you know we're adding more and more as envoy features come out. The only thing not in this dsl was the endpoints because that's supplied via the eds control plane, which I'll be talking about in the next slide, and we just have a regular kind of git ops flow of a pr it gets approved, merged and then deployed via something like jenkins into kubernetes.

A

So the um you know the control plane. We have right now we just implemented the minimum api and point discovery service. um We didn't, you know, implement the whole xds future set the endpoint discovery service was actually you know sufficient for us, it's built on, go control, plane and how it works. Overall. Just relatively simple, I think just kubernetes watch on pods on any new pods any deleted.

A

Pods change pods, the pod ips for each namespace are stored in memory on any deltas, we're sending out these new pods pod ips to the subscribed envoys, and then the egress clusters are defined in the config map, and then you know some some of the custom functionality we added in there was, uh you know, say: if an envoy and 1a az request endpoints, then it only received endpoints for 1a, so we were able to implement locality-based routing.

A

You know again just for saving on like cross az cost and latency um yeah, and then what we're working on this year is something called fire mesh. So we just you know like to go hard on the fire project names at tinder and we're implementing a full xts control plane with all discovery services implemented, so we're implementing uh on top of endpoint discovery, service route, discovery, service, cluster discovery, service, secret discovery, service and the listener discovery service trying to roll it out by end of year.

A

So if tinder goes down in december, you know what happened and then, of course we'll be piping. It all through ads the aggregated discovery service or using s3 for storage and kafka for messaging, and this unlocks, like header, based routing percentage based traffic, routing, which you can do without all these xds apis. But you know it's a lot.

B

More cumbersome.

A

Because you need to redeploy the pods or the cms and you know wait for those to propagate, but with the you know, xds, it's obviously very low touch, instant rollbacks and config changes um yeah, so just in summary, service owner is very happy. Everyone at tinder is very happy. We have this implemented. We have a holistic platform that integrates with the monitoring stack. The kubernetes stack, the ci cd stack.

A

I think the biggest win right is just immensely less code to maintain across the org from the network, automation, observability et cetera. We moved that all from the application layer to the envoy layer. I mean just some of the future work. We're going to be doing is ads, as I just talked about the delta xds onboard mobile for ios android.

A

uh You know investigate some custom, lua awesome filters and then do some request hedging, and that's it uh thanks for uh watching yeah. Do we have time for questions.

B

Yes, any questions.

B

uh So he asked us why we did our own control plane versus sdo. um The answer with that is uh we started building this before istio went 1.0.

B

um That was the real main uh thing of it, and once we got enough velocity, it was just too painful to switch over once we got our system fully pretty much where we wanted it, but yeah istio was great. You guys are just starting out. I fully recommend it.

B

B

Okay, cool: we will be on the slack if anyone has any questions for us there.

B