Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building a Multi Cluster/Env Service Mesh at Airbnb - Weibo He & Stephen Chan, Airbnb

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Building a Multi Cluster/Env Service Mesh at Airbnb - Weibo He & Stephen Chan, Airbnb

Tutorials and demos are great, but how do real organizations implement service meshes at scale? In this talk, we will discuss some of the problems Airbnb is solving with their service mesh based on Istio. Make sure you attend if you’re interested in building out a service mesh at your own company and interested in ways to adapt to your own requirements. We will walk through: - Partitioning workloads across multiple clusters and how to manage the mesh. - Testing mesh upgrades reliably with multiple environments. - Expanding the mesh to legacy, non-container workloads. - Routing traffic between regions, not just clusters, securely.

A

Hi everyone, I think, it's uh time to get started. It's 5 25., so welcome everyone to this session, building a multi-cluster environment service mesh at airbnb, and we have the presenters weibo he and stephen chan. My name is henrik blixt and I'm the moderator.

A

So just a few couple of notes before we get started so we'll do questions at the end. I'm gonna be running around with the microphone. So once we get to the q a raise your hand I'll run over as quick as I can with the microphone.

A

If there are questions we don't have time to cover. Please do that out in the hallway with the presenters, so we can clear the room and also, lastly, don't forget to rate the session in this, get scheduled app when we're done so just quick, intros and then we'll we'll get get going. So stephen is passionate about large-scale distributed systems, open source, technical leadership and engine excellence, and he's currently focused on solving scaling challenges of infrastructure.

A

High-Growth companies like airbnb weibo works on building performance, scalable and resilient distributed systems in the cloud and he's currently focused on building the next generation service. Mesh at airbnb, and will that, with that I'll leave the floor to the presenters. Thank you.

B

Thank you welcome everyone. My name is weibo, and this is stefan. We are both engineers from the cloud foundation team at airbnb. Today we are very excited to walk you through our experience of building a multi-cluster multi-environment service mesh on top of istio on airbnb.

B

Here's the agenda. For today we will start by introducing where we are at in our service mesh journey. Then we're going to talk about why we need a multi-cluster service mesh and how we deploy istio, to enable that, after that, we will cover multi-environment support, including multiple tier mesh, expansion and external services and finally, we're going to end with key takeaways.

B

In the past few years, airbnb, like many of our p companies, have transitioned from the monolith architecture to soa. At the same time, we also migrated the majority of our workloads from kubernetes from ec2 to kubernetes. As we underwent such fundamental infra changes. Our legacy in-house service mesh no longer meets our needs.

B

In 2019 we started the search of modern service mesh and landed on istio, as the foundation for more information about this choice feel free to check out our ecocom talk earlier this year linked on the slide. Last year, we evaluated different deployment models and deployed issue in production.

B

We started migrating a small percentage of airbnb workloads onto istio, and by end of last year we have about a few dozen services connected to istio and a few percentage of traffic going through istio proxy. This year. After successfully adoption on different kinds of workloads and operating, this is still in production. For a few quarters, we established enough confidence in both istio itself and our operational expertise in istio we started our full speed migration to istio.

B

Currently, we have connected almost all of our thousand micro services to istio and migrated about third of production traffic onto istio.

B

Our plan is to fully migrate and sunset our legacy service mesh next year, as we productionize our next generation service mesh, we encountered a few requirements.

B

The first requirement is multi-cluster, as airbnb doubled down on kubernetes adoption, our kubernetes usage exploded. We started running to scalability issues in scd and api servers, so instead of vertically scaling a single kubernetes clusters, we made the decision to horizontally scale out by distributing workloads across multiple clusters.

B

In practice, we keep each kubernetes cluster under a thousand nodes and add more clusters when we need capacity for more info about the multi-cluster design. Please check out our 2019 gupta talk linked below translating this into concrete requirements for our next gen service mesh. We needed to scale for tens of kubernetes clusters with hundreds of micro services, tens of hundred thousands of nodes and hundreds of thousands of pods.

B

How do we partition our workloads across clusters? At airbnb? We treat clusters as a pools of compute and memory resources. Workloads are randomly assigned to clusters at workload creation time. As a result, services makes cross-cluster network requests all the time. In this example, coaching three class of cross-cluster network calls were made. This simple example pretty much illustrated what's happening in production.

B

It often takes more than ten hubs, most of which are cross-clustered to serve a user request. This cross-cluster communication pattern heavily influences our next decision, as services communicate across clusters all the time, the added latency of cost of going through a gateway for every hub, it's too expensive for us. That's why we adopted a flat network model across clusters.

B

We leveraged aws vpc cni to assign individually addressable vpc ips to parts, so pods can make direct connection across clusters using vpc cni for cross-cluster communication has a few benefits over our in-house service mesh with you, which utilizes uh kubernetes no port services.

B

Vpcni has a lower overhead than no ports which use lots of complex ip table rules which doesn't scale particularly well for large clusters.

B

Vpc cni also does not lead to a well-known particle location problem, which is when multiple parts of the same service are scheduled onto the same node. They appear as one address and gets deduped by envoy. So each part on this. This node will get less traffic and finally, by providing parts with vpcips, we can leverage the reach functionalities. Cloud providers built examples include assigning parts. Custom security groups flat network does come with its requirements.

B

First, your organization should not solely relying on network boundary as the only security measure at airbnb. Our flat network is segregated into different security zones with security groups within the flat network. We follow a zero trust network model and um where all workloads are protected by mtls.

B

Secondly, a flat network requires all your workloads to be using non-overlapping ips. Luckily, for us this has always been the case. We, the infra team, centrally managed our private sideart ranges and allocated two slash, 10 side up blocks for mesh workloads which can fit 8 million ipv for my ips and give us enough headroom before we move to ipv6.

B

Finally, aws has a vpc limit on the maximum number of ips. You can fit into a flat network since vpc cni assigns unique parts, one node with one node with 16 parts consumed 17 ip mappings, which hugely increase the number of ip mappings in our vpc and push us over the limit. After close collaboration with aws, we now leverage a new vpc feature called prefix delegation instead of tracking part ips on the node separately.

B

The vpc now assigns a single slash, 28 prefix, which fits 16 ips by the way to a node and allocate part ips from that prefix range, which greatly reduce our mapping usage.

B

Taking into full consideration of all our multi-cluster requirements, we adopted the external control, plane and flat network model for istio deployments, which has the following benefits. First, it leads to a clean separation of rows. We, the service mesh team, serve as both the mesh operator and admin. We manage the upgrade and release of estio and we provide the service mesh platform.

B

Service owners, on the other hand, are in charge of their managing their own services, with tasks like defining, allow us which service allowed to talk to my service and allocating sufficient resource for their seo proxy. Secondly, external control plan provides better security. We only need to install the ca search issued by airbnb's internal ca system on the control plane cluster. We can enforce a tight access control for the control plane cluster.

B

Thirdly, um external control, plane model provides better isolation from data plane workloads. We want to make sure that no data plane workloads can affect the health of the control plane. Finally, we perform istio upgrades quarterly. It's much easier to operate, one control plane per mesh than end deployment across clusters.

B

A single external control plane does impose a strict requirement on the availability and reliability of the control plane. We will cover in the next section the steps we take to meet that requirement.

B

Here's our high-level architectural diagram within a single mesh we deploy istio in a dedicated external kubernetes cluster. It is the blue box in this graph. Only control, plane, components run from this cluster and only the service mesh team has permission to this cluster. All seo. Custom resources reside in this external cluster, but mesh class mesh users do not directly modify their resources.

B

We provide a simple configuration file that mesh users interact with they check in this configuration file alongside their service codes and during ci, we generate seo custom resources from that configuration file the generated seo resources are managed by our deployment system and all config changes are made by a deploy which is monitored and can be easily rolled back the remote workload cluster.

B

The the pink boxes in this graph only need to define a sidecar injection web hook, which delegates the injection of the istio proxy to the control plan cluster other than the web hook. Workload clusters only need to define seo reader row, which gives sdod permission to watch for kubernetes updates.

B

How do remote clusters dis discover the control plane? We run a deployment of external dns which, on the management, cluster, sorry and configure external dns to export ecld as a headless service registered in route 53 to be used as the discovery address for the workload clusters.

B

In addition to multi-cluster requirements, we also have multi-environment requirements for the mesh. The first kind of environment I will cover is multi-tier. We rely on a multi-tier mesh to minimize downtime of the control plane.

B

Airbnb services are generally divided into three tiers test: staging and production.

B

When building our next generation service mesh, we follow a similar concept of tiers to minimize the blast radius of changes. Our first tier of mesh deployment is the sandbox tier. In this year we run functional and performance tests to validate the new release of issio and isoproxy.

B

After the new release is verified on the sandbox tier, we deployed it to the test here. Test mesh has identical setup as the production mesh, but runs in a separate aws account for maximum isolation.

B

And, finally, we deploy a new issue release to the production mesh, both staging and production tier of services connect to production mesh in sandbox mesh. We run automated functional tests to verify mesh features that we depend on in production, like authorization, policy or locality based load balancing.

B

Additionally, we define regression tests for issues we found in the past. For instance, we encountered a bug where it still would get stuck if a large deployment was rapidly scaled down. We automated the repro of such regressions and make sure that issue continued to be fixed for future releases.

B

We also run data plane performance tests with our specific configuration of istio proxy using nighthawk in test mesh. We test every single integration point with airbnb systems and run end-to-end integration tests. For instance, we test that eco can secure, can securely mount the ca search from our internal ca system. We test new kubernetes version release, won't break the service mesh.

B

We test changes to the resource generation logic and finally, we can talk test all the custom logic we built for mesh expansion and external services.

B

We have successfully caught regressions in both istio itself and in related airbnb systems in test mesh.

B

After verifying the safety of the new release on the standalone mesh and test mesh, we deployed the new eco to a production mesh with a revision label. The initial deploy was safe as there will be no workloads connected to the new version. We gradually increase the scope of services connected to the new version, as we build confidence note that services only connect to the new version after a deploy at any time. If there is any regression service owners can simply roll back the deployment to pick up the old version.

B

The entire rollout process takes about a month after there's, no more proxies connected to the old version of the control plane. We will clean up the old version.

B

This rollout process allows us to slowly increase the number of proxies connected to old version. As a matter of fact, we have just finished up roll out of eco, 1.10 and teardown of 1.9 one month ago. This graph shows you a gradual increase of proxies connected to 1.10 versus 1.9.

B

We can also side by side compare the new version against the old version. This graph shows you the impact of how a new feature called discovery. Selector introduced in 1.10 reduced the cpu usage of istio when we excluded some of our most noisy namespaces.

B

Now I'm going to hand over to stefan and to talk about the rest of multi environments.

C

Thanks huevo, so we've covered how our service mesh spans, multiple kubernetes clusters and multiple environments. So now we're going to deep dive into two non-kubernetes use cases. First, we're going to talk about how we support workloads running on uncontainerized ec2, which internally we call mesh expansion as well as external services, like managed data stores.

C

So here's some context about why we wanted to support ec2 on istio. So in the long term we want to run all of our internal services on kubernetes.

C

However, in the shorter to medium term, we want all the benefits of a single uniform service mesh running across all workloads and the workloads that are not already running on kubernetes will take years to either migrate or deprecate. These are either legacy or very stateful workloads.

C

So the solution that we came up with was to support ec2 on our service mesh.

C

The requirements are first, we want to have full feature parity with kubernetes, so service owners can reuse their knowledge of how to work with service mesh on both ec2 and kubernetes, and secondly, we want to allow gradual traffic shifting onto kubernetes when services are ready to migrate.

C

So, let's go back and revisit. We both slide on kubernetes architecture.

C

This diagram adds a high level overview of how ec2 workloads fit in so the orange box on the bottom right shows ec2 workloads as part of the mesh.

C

Now that we've covered the high level architecture we'll dive into the mechanics of how we got to feature parity for ec2.

C

First, we'll talk a little bit about how endpoint registration evolved in kubernetes. The endpoints controller automatically handles this by updating endpoints resources based on pod readiness. The resources then get watched by sdod and distributed to clients.

C

When we first decided to adopt sdo, we had to manually tell the istio control plane, which ec2 ips belong to which services by hand editing configs due to the number of ec2 services in the instance, change update rate, this added unacceptable additional operational load. Over the past year, we've worked with the istio community to define our requirements.

C

The solution, the community built is a feature called auto registration, so the seod data, the istio data plane, has two processes which run on all service instances, whether ec2 or kubernetes. These are pilot agent and envoy. When envoy starts up, it will connect via xds to pilot agent, which proxies the connection.

C

Pilot agent will then connect to the sdod control plane and provide metadata about which service it belongs to using that xds connection, sdod will create a kubernetes crd called workload entry on behalf of the connecting pilot agent process and then the workload entry is what registers the service instance in the mesh.

C

Other sdod pods will read workload entry and distribute that service instance to other clients.

C

Server-Side health checks were also implemented by seo community in the past year. The health check process begins in the same way as auto registration.

C

First pilot agent proxies xds for envoy pilot agent, then does health checks, which is similar to how cubelet does health checks in kubernetes pilot agent will then send health information across the same single xds connection to an sdot control, plane, pod and seod will alter the healthy status of autoregistered workload. Entry.

C

Other sdod pods will read and react accordingly, adding the endpoint if the workload entry transitioned to healthy and removing it if it transitioned to unhealthy.

C

The tls story is a little bit more complex, so istio community plans some automation around getting certificates, but the timeline is unknown. So we built our own automation. There are two tasks needed to boost drop tls.

C

First, we need to get the root ca and second, we need to provide proof the ca to get a workload certificate with a spiffy sand.

C

Sdod starts by writing the root ca as a config map in the kubernetes api, we built an in-house tool called mxagent which creates a service account token and reads the ca. Config map that sdod wrote by calling the kubernetes api mx agent then writes the ci cert and token to the local file system. On the vm instance, when pilot agent starts up, it will read the ca and token from the file system and then issue a csr to sdod.

C

Finally, envoy will get its certificates from pilot agent using sds.

C

Seo version upgrades are also slightly complex, so let's backtrack and quickly review how istio versions are chosen in kubernetes. So in kubernetes we use an internal tool called crispr, which runs as an admission controller.

C

It reads: a configuration file called rollout.yml which maps kubernetes namespaces to mutator artifact versions, and we wrote a blog post about crispr, which is linked in the bottom of the slide: crispr downloads, the mutated artifact from storage and then patches, the pod label, with an istio version label seod then gets called as another mutating admission controller, which does the main sidecar injection and the sdod version called, is determined by the istio revision label.

C

Crispr is designed with the kubernetes use case in mind, so we couldn't reuse this for ec2. However, we took key lessons from crispr and created a very similar interface for ec2. We built a parallel system. First there's a cli named mx rollout, which reads a rollout file similar to crispr's rollouts configuration format.

C

Mx rollout will then update an ec2 tag named desired version. On a particular instance, when mx agent starts up, it will read the desired version, tag value and this tag value maps to an artifact which contains pilot agent and envoy binaries, as well as other configuration mx agent, will download this artifact unpack it and restart the pilot agent service running on the host.

C

Finally, mx agent updates the value of the running version tag on the instance: here's a high level overview of christopher and our mx updating system side by side. There are several similarities that you can see between them: okay, let's switch tracks and talk about how we supported external services.

C

Most of our external services used custom protocols like mysql and redis. So when scoping solutions, we grouped external services and non-http services together.

C

External services can't run istio data plane components, so we have a slightly different set of requirements compared to supporting ec2. First non-http support was important. We wanted to allow service owners to define custom, redis, pings or mysql health checks. Secondly, we wanted to use dns to map external service urls to a unique kubernetes cluster. Ip external services are registered in istio using service entry, but without an ip field.

C

Service entry results in an envoy listener that takes up all ips on a given port, we're moving off of a service discovery system that relies on port allocation when we want to avoid this kind of setup wherever possible.

C

And finally, we want to remove stale endpoints, for example, when database replicas are torn down.

C

So here's how we solved server-side health checks, we built a generic health check, sidecar called mhcx airbnb internal owners will build protocol specific health checks into an application container and then run mhcx as a sidecar.

C

Mrcx is responsible for translating healthcheck results into workload. Entry updates effectively doing the same thing that sdod was doing with vm health checking and then the app health checker main container is responsible for querying external endpoints, using the custom protocols and providing an http response to mhcx.

C

For our dns cluster ip setup and endpoint removal, we have two parts so first for the dns cluster ip setup under the hood, we create both a kubernetes service and a service entry. Crd.

C

We ensure that the service is created first before the service entry and then when a new service is created.

C

When the new service entry is created, we add a mutating admission controller, which patches the service entry uh ip with the cluster ip of the generated service and for stale endpoint removal. We rely on spinnaker to run a cleanup command, which is similar to keep ctl print.

C

Now that we've covered several environments in which we run seo, let's cover the key concepts which we use to successfully run it first, we ran a globally flat network but be aware that there are considerations around utilization and scalability of this approach.

C

Secondly, we use a single external management cluster for each service mesh. This is important for secure and simple management of each mesh.

C

Third, we use multiple tiers of pre-production meshes which allow for different types of testing and ensure we catch as many issues as possible before production.

C

Fourth, istio support for vms has matured considerably. In the past year, we built additional tooling to reach full feature parity and allow a long-term migration path to kubernetes.

C

And finally, we have generic mechanisms for supporting non-http and external services, which ensures the service mesh team does not have to implement any different envoy filters for n different protocols all right. Finally, airbnb is hiring, so please check out our website for open positions, and now I think we have a few minutes for questions.

A

Thanks looks like we have about eight minutes left for questions, so raise your hand, I'll try and run around with the microphone starting up front here.

D

uh Hi, that was a great presentation. You had a lot of things packed in there one question uh when it comes to a lot of services in the cluster that istio has to look upon. So I know istio gets a lot of charity uh chattiness. How did you manage that.

B

Yeah, so istio has a sidecar crd that filters out the listeners, the clusters that you get as part of our simple configuration file that we we ask service owners to provide. We like force everyone to define the exact set of uh other services that they care about. So that way, it's greatly reduced. The number of um like unnecessary updates to each service.

B

A ton of services and different names becomes totally understood. We tested uh seo without uh using like that site card crd and the performance it's hard to scale, for, like tens of thousands of thousands, thousands of services.

E

Again, I was more curious because our environment looks a lot like your guys's. um Have you tried the multi-cluster mode in istio and not use the service entry.

C

Can can you provide a little more information about like how the service and how multi-cluster would um yeah.

E

So the newer versions of seo you can define different, you can call your um different seo, setups different names um and then basically, it talks to the kubernetes api um that is trying to access and then the the service endpoints are advertised everywhere. So you don't have to create service entries there. All clusters know about all the endpoints.

C

Oh, I see so we are allowing sdod to read from multiple clusters for service entry. We use that exclusively for external services, so, like services that we aren't running in-house.

B

So yeah, if we go back to the architecture diagram, we are running multi-cluster mode, so the centralized seod subscribe to different workload, kubernetes clusters and for kubernetes workloads. They use directly use kubernetes services and kubernetes endpoints and uh we don't define the service entry. It's only for non-kubernetes, like vm, workflows and external services. We define service entry because there's no pods, okay,.

E

And can I ask one more question just curious, um because we again we run into this problem, do you guys use the helm, chart or use that the profile is your profile file? The crd.

B

uh We use the operator to generate uh the the basically raw manifest and check them in into the source code. So we know what like uh we, we feel uncomfortable uh asking it operator to handle all the upgrades. We would like to know what exactly is going on with each eco release.

B

A

um Yes, where does the operator that uh injects the sidecar lid? Is it.

F

In the control plane cluster, or is it running in each.

B

Individual cluster, so the question is: where does the sidecar injector live? The web hook live on every workload cluster, but the mutator service. The ecod only runs from the management cluster, the external cluster, so there will be a per workload cluster web hook that basically points uh to the central uh ecld address.

B

Since you bring up so many clusters.

F

Dynamically, do you ever hit any weird race conditions where the web hook is maybe not up before the application caught and then your sidecar isn't alive or something like that.

B

C

Yeah we run uh like verification workloads on each cluster before we mark it as schedulable, basically before we can assign workloads to that cluster um and our rate of cluster creation is like more on that, like order of hours rather than like minutes or something, so we usually go through a verification process.

F

B

G

uh One one of the most painful things uh we face when we upgrade hto uh version right is the envy filters, because the apis break really easily. It is super painful. How do you manage that in airbnb.

C

I mean yeah yeah envoy.

C

We we try to keep our envoy filters fairly portable, like we've written two, that one that customizes logging slightly and one that customizes metrics to use like a native envoy filter, but um definitely agree that um there are some pain points with with upgrading like because the like internals of, like envoy configuration are unstable. So we just try to keep it as simple as possible. There.

B

Yeah, we definitely felt a pain point.

H

Hi yeah, I had a question um you mentioned earlier that you noticed that you had a bug where istio was getting stuck on rapid scale down of deployments. I was wondering if you could provide a little more details and like where was istio getting stuck in how you fix that issue.

B

Yeah um that particular issue uh like I can't send you the get the issue offline. I it's it happens a while ago. um I think um what we observe it's when uh large deployments rapidly scale down and sld will basically fail to send out any subsequent updates, um like more details can be found in the github issue.

B

I don't remember any more details right now. Sorry.

C

The the issue that we ran into was, I think, in 1.5 or 1.6.

B

C

Several minor releases ago- and we haven't observed it since.

F

So when you have two over here, when you have two clusters um can uh uh when services in one cluster, can they talk to services? The other cluster without, like changing anything about the service definition itself was uh like so the services themselves wouldn't know which cluster pods they're talking to, or you have to like, specify which cluster you want to talk to when you.

B

So when services services, when, when they're making cross-cluster network costs, they are unaware of the destination cluster, so um we define a mesh id like us depending on the region, ea, one us and that's shared across clusters. So from the perspective of a service, it doesn't know where the destination service runs from.

B

As a matter of fact, that service can run from multiple different clusters and sdld is going to combine the endpoints of like fetch the endpoints and combine the endpoints uh group, the endpoints under the same namespace and so requests can be load balanced across clusters.

A

So I think we're we're at time. So so thanks thanks er, we can uh continue uh questions outside in the hallway, so we can clear the room so they can close down. So thanks everyone for coming appreciate, it don't forget to rate the session afterwards in the in.

H