Cloud Native Computing Foundation EnvoyCon 2020 - Virtual, 12 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How Tinder implemented Envoy global rate limiting at scaleVirtual - Yuki Sawa

Description

How Tinder implemented Envoy global rate limiting at scale
Virtual - Yuki Sawa

Tinder recently completed a migration to Envoy based service mesh in their Kubernetes based infrastructure. A big win was moving rate limiting logic out of the application and into the network layer by leveraging Envoy's powerful global rate limiting capabilities. Previously implementations relied on home-grown code inside the application or features built into proxies like Nginx — which were difficult to maintain and did not offer the configurability and observability of Envoy. This talk covers how Envoy global rate limiting works at Tinder, how we migrated to it, and what steps were taken to ensure it performs at scale. We'll also discuss the unique rate limiting features available in Envoy, how to configure it and how we extended upon it.

A

Hi, I'm yuki and today I'll be talking about how tinder implemented envoy global rate limiting at scale. I'm a software engineer at tinder in the cloud infrastructure team and in my day-to-day I work on kubernetes and envoy, so in particular, I've implemented the xts control plane for our service mesh, which all our envoys talk to, and today I'm going to be going into detail about the rate limiting platform I built there, which is also based on envoy.

A

So in summary, I'll, be talking about these things. First of all, our service mesh journey, how we implemented envoy across all our services, the problems with our previous rate, limiting implementations envoy rate milling, why it's good, how envoy rate limiting works in detail, how tinder's rate limiting works? How we've extended upon it and just some tips on if you want to try it out yourself, so tinder's service mesh journey. We were fully on kubernetes before starting a service method migration.

A

So we started the service mesh migration about a year ago in 2019 it took about a full year for all our services fully to be on envoy. It was definitely a very big effort, took many people in the organization to achieve it, and today about 1.6 million requests per second are meshed. You know in our service mesh at peak and there's about 200 000 containers in our service mesh.

A

So this is just a chart of how our infrastructure works on the left is a user. It's using the tinder app. The request gets routed to one of our three clusters, so we have a cluster per availability zone in aws, there's dns round robin that routes, the request to one of these clusters and each cluster is isolated, so they do not talk to each other.

A

This gives us a lot of like benefits in terms of performance as well as, like traffic bandwidth cost savings, we have an ingress layer which used to be nginx, but now we might get the envoy and once the request makes it into a kubernetes cluster. Of course the request is routed to the correct service.

A

So we tried out envoy because initially the default kubernetes elb routing was not that great. It was very uneven and resulted in uh hotpot issues where pods some pods got a lot more requests than others, but eventually you know we started trying out a lot of the other great features envoy has like these requests. Routing. First of all, right, uh there's retries circuit breakers timeouts that you essentially get for free, now, you're not having to do all this network automation inside the application.

A

It's all on the envoy network layer. Now we're exploring like the redis full red filter, dynamodb filter, where you're proxying database requests through envoy. That way you get like metrics for free uh now, we're uh of course we're doing observability where all our envoy metrics are being scraped through prometheus and then charted on grafana, so previous rate, limiting implementations, a tinder used nginx at the ingress layer. Problem with this is uh there was really no visibility.

A

You would need to like tell logs to figure out if the rate limiting was even working or you know, what's being rate limited. um So that's not ideal. It was all done locally, as in it was based on the number of requests. uh Certain nginx hosts saw not on the on the global request count. It was also very difficult to update the rate limits because you need to you know, roll all, the nginx hosts and then. Secondly, there were implementations inside the application, so a lot of teams would like roll their own rate limiting code.

A

So there's a lot of different implementations across the org there's little visibility into how they were working or how they were configured and often there was redundant infrastructure.

A

You know, because a lot of teams would use reddish caches for the rate limiting and there were duplicates of those basically serving the same purpose. So why is envoy rate limiting good? We were able to move all the rate limiting logic to the envoy layer, so we have a uniform implementation across the company. We have global rate, limiting where the rate limit is based on a global request count, and this is really important for our cluster per ac model.

A

It's got granular configuration such as you can rate limit on multiple headers or one header, or you know, even no headers. You have monitoring and visibility through the prometheus metrics. It offers and overall we're able to prevent system failure, patch concurrency issues and, due to you, know stopping like spot traffic. We have a lot of cost savings and right now it polices about you, know: 200 000 requests per second at tinder, so this is a chart of onward rate, limiting it's a request flow of service, a making a request to service b.

A

So, let's start at one a makes: a request to b and service a has an envoy sidecar right. So the envoy sidecar asks the rate limit service. Should we rate limit this request or not? If the answer is yes, a 429 status code is returned to the service a container.

A

Otherwise the request is let through and a successfully makes a request to b, and you can notice here that on the very right, the rate limit service is uh storing all the request count: information in a redis cache.

A

Okay. So let's talk about the rate limit service. It is a go project and deploy it in kubernetes pause with an envoy site card, and this way we can proxy requests to redis through envoy, which gives us metrics for free, as well as being able to tweak some redis connection settings.

A

We have two separate rate limiting clusters, one for internal routes, another for external routes, and then you might be wondering what happens if the rate from the service is down or if it's slowing down there is a 20 millisecond default timeout to request to the rate limit service. If it is exceeded, there is a fail open, meaning that the requests will be allowed through by default and what's nice is that, just by updating the rate limit service comic map, it is hot reloaded.

A

So, if you're changing from say, like uh rate limit from five requests per second to 10, requests per second just update the config map and it will reload automatically.

A

So let's look at the configuration it's important to note all the envoy configuration is done in the caller, so here with this route, you're attaching a descriptor key called foo.

A

If that request has a header name of foo, if it has a request with a header called foo that is, and then with that descriptor, it is sent to the rate limit service and here's the corresponding weight limit service config.

A

You can see here that it has a nested structure which allows for pretty complex uh rate, limiting logic and then here for the food key. There is a corresponding two weeks. Two requests per minute specified. So in the rate limit service, config is where you define the thresholds.

A

So let's talk about one use case at tinder, which is sms, request rate limiting. So what happens? Is we have a lot of bots requesting a lot of sms codes, which is very expensive because twilios is twilio is expensive, which is our third-party sms provider?

A

And initially we did have this lately winning built into our applications, but it was brittle hard to update every time you want to add a new rate limit. You'd have to write. Additional code was not ideal, so we migrated all of it to envoy which had uh millions of dollars in savings, and we had these very adaptive rate limits. So for each ip it had a per day and a per second quota and we could rate limit on a combination of headers.

A

So if one user had an ip of one one one one, the country is us devices ios, they would have like a five request per second rate limit, while if they had a different ip address their country is in japan. The devices is a web, they would have a lower rate limit.

A

So on top of this, we also built in a analytics module into the rate limit service. So every time there is a rate limit event, it is sent to s3 for processing and long term storage. So what we can do is automated block listing. So if, like a user was rate limited three days in a row, we may ban them, we may ban them for like 30 days uh and now we can also analyze long-term behavior.

A

We can, you know, say you know how did the rate limiting perform like 90 days ago, who got rate limited on what routes? Now we're also starting to be like machine learning on this data to enrich our existing bot detection? So we're finding a lot of value in this weight limiting data, so this is just a cool little chart I made so I got all the requests that got rate limited, got their geo locational data and put it on this world map.

A

You can see that there's really bots coming from everywhere, particularly around where aws data centers are located. As you can see, there's a couple big dots in virginia, where aws usc 1 is and then just. Lastly, some tips. I would scale your infrastructure so that your p99 9, latency to the rate limit service is 20 milliseconds and not much more than that. Otherwise, you're going to have a lot of fail, opens 5, 000, quests per second per rate in the service pod is a pretty good benchmark.

A

I would use the envoy redis filter to make your life a lot easier, particularly because this gives you redis performance metrics and how redis performs is going to affect how your p99 latency is. So if you want to just try it out on your laptop clone, this repo run this docker compose command and you can just curl your local host and you should get a 429 response like you see here on the bottom right.

A

Thank you. That was all shoot me. Some questions on twitter if you got any, but thanks for listening.