Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Service Mesh at Scale: How Xbox Cloud Gaming Secures 22k Pods with Linkerd - Christopher Voss

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Service Mesh at Scale: How Xbox Cloud Gaming Secures 22k Pods with Linkerd - Christopher Voss, Microsoft

During this session, Chris Voss, Senior Software Engineer at Microsoft, will share how Microsoft deployed Linkerd to apply mutual TLS, observability, and reliability to 22,000 meshed pods across 26 clusters. Xbox Cloud Gaming is Microsoft's game streaming service. With over 300 games available in 26 countries. The app is massive: 26 clusters across 18 regions, each with 50+ microservices and 700 to 1,000 pods — all of which are meshed with Linkerd. Chris will cover Xbox Cloud Gaming's Kubernetes and Linkerd journey, including how they: * Applied mutual TLS to 22k pods with zero config * Reduced pod/container monitoring cost by thousands of dollars using Linkerd observability * Integrated Prometheus, Linkerd, Flagger and Azure ADO for progressive delivery.

A

I'm chris voss, that's a picture of me. If you can't see me from the back, um I'm a senior software development engineer at microsoft, I joined both microsoft and xbox cloud gaming in 2018 august or so originally I was uh on the team that is managing our xbox servers, um keeping them healthy, making sure they're up to date all that fun stuff. um More recently, last couple years I've been on our infrastructure team um game game streaming, shared services.

A

Our team manages and operates the infrastructure for xbox cloud gaming services. So you know azure resources such as cosmos, db or storage accounts or key vaults, all that fun stuff, our kubernetes clusters. Obviously, and then we also have some services that are kind of shared across all of our other teams that we also own.

A

If you'd like to contact me, these certainly work. I'm sure there are others that I forgot, but you know if you want to reach out to me. These are definitely good places, so a little bit about xbox cloud gaming. So um it comes for free if you have xbox game pass ultimate and basically it enables you to play games wherever you want on whatever device you want and our motto is or our mission rather is allowing people to play the games. You want with the people you want on the devices you already own.

A

You know it's important uh to us. um You know to enable anyone who would like to play games to play those games. It's you know it's a way of connecting people who might be you know from great distances and giving them shared experiences and um so yeah. That's a big motivation behind our team and our product.

A

um So a little bit about what our footprint looks like we have 26 roughly aks clusters and aks is azure kubernetes services. I forgot to add that in there, but across several azure regions we have 50 or so micro services. I think it's a little bit more than that now, but 50 plus is good and about 700 to 1000 pods per cluster, so yeah spread spread across those 26-ish clusters.

A

um We have about 22 000 pods around the world um and and just a little bit about the servers you know so when you're playing a game um you're streaming a game.

A

um What you're streaming from is actually a xbox series x hardware, um it's a custom design, modification of the xbox series x, hardware and it's deployed in data centers around the globe, so when you're playing you're really playing on an xbox and then we're giving you all the signals um and uh allowing and then allowing your feedback to come back and connect to that console.

A

So a little bit about our tech stack, um you know. Obviously you know we're we're part of microsoft, so we're going to be using microsoft products, but we are also huge adopters of as much open source as we can. These are some cncf projects, um along with uh some of the other stuff we use. So uh you know all of our services are written in dot. Well, the majority of our services are written in.net, we run inside azure, kubernetes service.

A

We use docker to containerize all of our services and we use helm to deploy them and we use linker d along with flagger. In order to use canary deployments and I'll talk more about that later and fluent bit is what we use for our logging pipeline and prometheus, as you might imagine, is metrics.

A

So you know to kind of level set make sure you know, I'm not sure how many folks are already completely familiar with service meshes, um but you know just make sure we all are kind of on the same page. You know what is the service mesh um and, of course you know there are several different service meshes out there.

A

All with you know unique functionality, but there are some things that kind of unite them as well, and so um really the way I think about it is you know it provides controls around the traffic inside of your clusters, sometimes out of, um but you know for the most part it's inside that cluster and there is a spec for the service mesh interface and it has specs around traffic access control.

A

Metrics, specs and specs are really like the crds for routing rules and such and split, which is what we use in order to do our canary releases, and so, if you want to check out the the actual full spec, you can go to that link all right. So you know now we understand at least somewhat what a service mesh is. uh You know why do you want one? Why did we want one?

A

Why might you for us a huge huge, huge motivator was to simplify our service to service tls or mutual tls mtls for short story. Previously we managed our own solution.

A

We had to you know, get our certificates created, get them deployed into the right clusters into the right name: spaces all that fun stuff load them into our service, and then we got the fun option of troubleshooting it when there were problems and that caused a lot of pain in the beginning and so service meshes became on our road or you know on our horizon pretty quickly because it created a lot of friction of like spinning up a new service or things like that.

A

We also really really really wanted code flighting and progressive deployments we wanted to be able to. You know, build confidence in a deployment before we just shove it on out there on everyone. We've had several times where that saved us from like major outages, because um our uh auto roll back uh worked um observability. So you know just like. In the the service mesh interface spec, there was metrics, you know, being able to understand.

A

What's going on inside your system inside your cluster inside your service, mesh incredibly important, as you probably know, and most service meshes, provide that for you um and code free instrumentation that was huge. um We did not want to. You know, have to pull in tons of different libraries into our net and hope, they're in.net, and then um you know bloat up essentially our containers with code that we weren't necessarily maintaining.

A

So you know our service mesh search began around 2020.

A

So basically, what happened was a few of my colleagues went to kubecon cloud native con 2019 in san diego, and uh specifically my my former manager um saw a linker d um booth and there was like a challenge to set up mtls in a cluster in five minutes and he was able to do it and so immediately that put link or d on our list of like hey those are service meshes, you should check out.

A

Obviously you know it's cool that like one was easy to use, but you want to make sure you're still choosing the right one, and so um you know we we evaluated several which are listed here.

A

We we did want to make sure you know. Every service mesh met our requirements, and so we wanted to make sure it uh implemented the service mesh interface because we wanted to use canary deployments. We also wanted to make sure we had efficient resource utilization, especially cpu, because most of our services are pretty much cpu bound, and so you know any service mesh that we're adding. On top of that is just going to add more cpu pressure.

A

If it's high utilization observability, you know I already talked about that in the previous one, um but we need we wanted to make sure we could. You know, make sense of what was going on inside the the service mesh and set up slash, maintenance ease, and you know this is really we don't want to have a team, that's dedicated to our service mesh.

A

We want to have a team, that's dedicated to all of our infrastructure, so if the maintenance or or setup are incredibly difficult and painful, we'll likely have to dedicate you know one two people to it, just to maintain it over time. That was just not scalable for us.

A

So we we sat down and we said, okay, you know we kind of looked at all the features and said: okay. These are the few that we really want to investigate and so like someone on our team put together prototypes and we evaluated all of them and, as you might imagine, linker d1 winning is maybe not the right term. It's more fit our needs exactly so.

A

You know basically efficient resource utilization. It was very good there. Traffic splitting you know using the service mesh interface um we're able to get that functionality. Observability comes with a ton of metrics out of the box and low latency. um You know, obviously, whenever you're adding something into your call stack. If it is, you know adding a significant amount of latency and you have multiple layers of calls that can get pretty nasty, pretty quick, and so um that was a big deal for us as well.

A

So you know some of the ways that we're using link d um to scale our business. So now that we've chosen linker d, um you know we're we're using mtls, obviously with cert rotation, um which you know it's in the title of the the talk. So that's probably not super surprising um we're also using it in high availability mode. This is something that we missed kind of. When we were initially doing our deployments. Is um we saw that like if a bunch of services were getting deployed?

A

At the same time, they were having um a container uh yeah pod injection. uh Sorry container injection into the pod um was having issues essentially and that's because there was only one instance of linker d running and we were not running in high availability mode. Pretty easy thing to fix just was something we didn't recognize right away very important if you're going to production with lincoln prometheus metrics. So to be completely honest, we did not use prometheus metrics.

A

Before we started using linker d, we had a way of getting some of the internal metrics out into our own systems, and you know we used that, but it was costly and again it required a lot of maintenance from us.

A

Obviously, we use the service mesh interface extension for canary releases, because it's very important to us to have that and linker does support that also- and this you know is probably something that can be said about. You know most open source communities, but you know this is one of my first experiences with it, and we've had a really good experience working with the lincoln d community. This link here is a colleague of mine, abraham.

A

He was the one who kind of went through and did some of the prototyping work and opened several issues on the linker d github and you know, worked with them to drive those to conclusion in order to enable us to uh move forward so I'll go into a little all of those in a little bit more detail, so mtls zero, config mtls is simple um and honestly I considered having that be the only bullet point on this slide, because that is how much most of us have to think about it.

A

Now that it's all set up, it's essentially um like so, for instance, when uh we have been moving to a new architecture for our clusters, and um I was tasked with kind of getting all of our uh uh plugins to kubernetes. Working and linkerd is one of them, and I was incredibly nervous. I'm not a security expert, I'm not a certificate expert or anything like that, and when I put down what I thought was the right things and um you know executed our terraform code.

A

I was confident that I had messed it up because it just worked and to be completely honest, I had to ask other people to like help me figure out whether or not it was working, because I really really didn't believe it, but it honestly was that I mean that is not a lie like that's you know. Sometimes people say things like that, but it genuinely was. I had to go and ask uh someone else to just help me check my work because it seemed too easy.

A

You know- and you know, like I said previously- we've secured over 50 micro services and 22 000 pods around the globe so observability. This was huge for us code, free visibility, so we don't didn't need to have a bunch of instrumented code in order to get these metrics out.

A

We also, interestingly, essentially used some of the metrics to drive our canary deployments. So I'll talk about that a little bit more later.

A

So maybe I won't jump the gun there, but um you know so some of the things that we we use it for http response codes, latency monitoring, obviously canary deployment status, request volume and it comes with its own prometheus, which for us again, you know we weren't using prometheus before, and so the fact that it came with its own prometheus instance um really kind of allowed us to see the power of what we were missing, and so it also pushed us to kind of adopt that um so, let's dig in on canary deployments a little bit um so for folks who are not familiar with canary deployments.

A

They're similar. You know in idea to like green blue deployments, but slightly different, green blue is where you like stand up. You know essentially a duplicate of your service, and then you tear down your previous one and it's essentially a seamless cut over for canary deployments. Though we have this little visualization, uh you can't see my mouse, but um we have this little visualization. ah Here we go laser pointer all right, so we have this little uh visualization here. So you know in stage one when I'm, you know deploying a service.

A

uh You know these controllers represent people who are wanting to play a game right, and so you know initially we're trying to be cautious, and so maybe we'll give one out of six gamers a uh you know, access to our new new deployment, and if we find that that's working well, we eventually move it on to um our second stage of you know two control, two players out of six and you know further on the more confidence we build uh the further along it will deploy and we'll either detect.

A

You know some sort of issue with you know. Perhaps it's a scale issue. You know. Maybe it gets 75 percent of the way or 75 percent of the traffic and then all of a sudden, there's a scale problem and we roll it back. We have integrated our canary deployments into our azure devop release pipelines to enable auto roll back. So if we, if we detect that, there's a problem, we will automatically roll roll back.

A

So um you know in that instance, it would go all blue controllers um or in the uh case of success, which is you know, hopefully the more likely uh scenario. uh You're gonna have all green controllers, so everything is executing new code and it's a flexible canary evaluation.

A

So some of our key learnings from flagger things we didn't think about when we first set out trying to use it. um The biggest thing was how to handle canary deployments for different types of workloads.

A

You know, and some of those are like non-http calls uh you know, such as like udp or things like that or asynchronous call patterns. You know like you're, using a message queue or things like that or cron job type workloads. We have several services that have like periodic background services. That kind of spin up do some work inside the service. Maybe you know, fill a cache or something like that and then go away. You know detecting whether or not those things are you know.

A

Good in a canary scenario is not just looking at http response codes or you know latency, but instead emitting some sort of signal that gives your uh you know canary evaluator, a um idea of what's going on uh and also not all, services have constant or high volumes.

A

So, like sure, even if your service is entirely bound to you know, receive http calls if you're in a dip in your you know daily cycle or whatever you may have some issues with uh false positives or maybe under reporting things like that, where you're not able to make a um a super, great evaluation- and this is something that really was sneaky for us um traffic to our liveness and readiness endpoints. We were skewing our canary evaluation, um so in the instance on those like you know, dips and things like that.

A

If we were deploying- and we saw you know- oh, it looks like there's a fairly steady call pattern across. That time looks good to us, but really all it was was our health endpoints, which absolutely are important and, hopefully are you know indicating some level of health. But you know, some of the point of the canary uh deployments is to detect the things that you aren't already checking for, and so that's just another thing to you know keep in the back your mind.

A

uh If should you go forward doing something like this all right, so engineering and cost savings? This is huge and not something that we necessarily expected. We hoped for, but didn't necessarily expect um so engineers were freed from supporting in-house mtls. Not only is this like a time savings and engineering work, but it's also a happiness improvement because not everybody wants to have to think about. You know how do I get my cert into my service and into my cluster and how do I get it generated and who do I have to go to?

A

You know to rotate rotated or things like that, and so you know happiness time awesome uh and we also had reduced, spend on alternative, alternative, observability solutions, and you know I had mentioned.

A

I think briefly previously that we were kind of sending some of our kubernetes metrics out into our own system and and handling it there and kind of coalescing on prometheus for all of our kubernetes metrics have really helped us reduce, spend there, and these two things alone have resulted in thousands of dollars of savings per month, all right so a little bit about where we're going to be going with linker d. These are things we haven't fully fleshed out their ideas.

A

We've got they're, you know, scenarios on our backlog um but uh service to service off. So you know sure we've got mtls. We've got secure communication, but you know if a bad actor gets access to one of our pods and they start. You know making calls to other pods to try to find you know, vulnerabilities or things like that. We want to try to reduce the risk of that and linker d does support that um also multi-cluster communication and failover. So um previously, in our previous architecture, um we weren't really.

A

We had no need for multi-class communication. All of our clusters were essentially, you know the same, and they you know had their perfect scope and they didn't really care what was going on in the others, which was convenient for some situations, but not great for other situations, especially reliability, and so now that we've got kind of a better architecture for it, we're absolutely going to be working on multi-cluster communication and definitely failover. So looking forward to that also fault injection and chaos, testing things that you know you can essentially fake.

A

You know: oh what happens if all of a sudden, my pod stops being able to talk to any other pods or things like that.

A

I have very little experience in that space. So I'm really looking forward to learning more about that all right, I'm almost done, but I wanted to make sure um before I wrap up. I wanted to introduce everyone to my colleague, abraham modeiji.

A

He was unable to make it to valencia, but he's an awesome dude and he did an incredible amount of work in our path to consuming and using linker d in a production. And I wanted to make sure that his work was not missed by me. Presenting because he's absolutely an integral and very important uh part of the equation.

A

And thanks for joining, if you have any questions, I'm happy to answer them um and if you're interested in trying xbox cloud gaming, you can go to xbox.com play, you can play fortnite for free um or you know, if you're an xbox game pass ultimate subscriber. You can play hundreds more thanks. A lot.