Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Pushing the Limits of Kubernetes with Game of Thrones - Zihao Yu & Illya Chekrygin, HBO

Description

Keynote: Pushing the Limits of Kubernetes with Game of Thrones - Zihao Yu & Illya Chekrygin, HBO

Do you want to know what it is like to run 15,000 pods in production? Are you interested in seeing how Kubernetes stands up to the record-breaking viewership and a login rate that is beyond belief on Game of Thrones Season 7 premiere? Come and see things we have done for the Game of Thrones preparation. We will talk about how we provision Kubernetes clusters on AWS, and how we monitor them and microservices that are running on the clusters.

In this talk, we will also go over how HBO Go went from deploying and running microservices on virtual machines in AWS EC2 to running the very same services inside the Kubernetes clusters. We were able to dramatically increase the productivity of our engineering teams and efficiency of resource utilization in the process. It wasn’t always a smooth ride and it wasn’t a one shot deal. Instead, it was a long and at times challenging journey starting from operating a reliable, production-ready Kubernetes cluster in AWS, advancing to gradually deploying select services into Kubernetes clusters, load testing them, and running them in parallel to our current EC2 installations, and finally going live. Come and learn some helpful tips and mistakes we made along the way, which could help your organization embrace the Kubernetes world.

About Illya Chekrygin
Illya has been working on Kubernetes adoption at HBO, which includes cluster provisioning, maintenance, telemetry and service migration. He also drove the containerization of HBO's core streaming services and CI/CD integration for their traditional EC2 deployments. Prior to HBO, Illya was using Kubernetes at Zulily, Inc.

About Zihao Yu
Zihao Yu is a Senior Staff Engineer at HBO, helping HBO GO backend services deploy faster and more reliably. He has contributed to the design and development of several iterations of cloud infrastructure and CICD pipelines for deploying microservices at HBO. He is currently working on the operationalization of Kubernetes, including provision and upgrade automation among other DevOps topics.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Let me say it's a huge owner and let me first thank the Academy hey wrong stage. All right, my name is Ilya. This is Z, and today we'll talk to you about HP journey to kubernetes the journey which began not that long ago, from not having a single service running inside a container to hosting game from season. Seven on kubernetes, Z and I will put this talk. I'll tell you about why reasons and Z will cover how we get it done.

A

Okay, it's donkey from there. It's boys come to the shows that everyone is talking about from the groundbreaking series to the documentary sports to the biggest blockbuster movies available anywhere for over 40 years. People who love entertainment have recognized HBO as the original the first and the best place to find world most innovative programming. Hbo digital products is represented by HBO GO, which is part of U TV programming subscriptions through cable, satellite or other providers, and HBO now subscription directly with HBO. Both provide unlimited access to HBO programming just about on any device.

A

It brought sense. Digital products is everything anything to do with the content streaming and user products, that's where Z and I work and platform team.

A

So if we zoom, if we look at the HBO streaming services, this is how they could look like from the mile-high view. This is not the actual image, but resemblance is pretty close, and if we zoom in they could be best described as a matter of API services written mostly in OGS.

A

Also, we added more and more sources written ago and all sources were deployed into ec2 into single service instance single ec2 instance paradigm, all wrapped with auto scaling group which handle both deployment and scale all fronted with the internal or external load balancers and route route. We handled a DNS need, overall, it was, and still is, tried-and-true set up for running services and edible years. It works for general case.

A

However, if you will see next, that HBO case is anything but general HBO traffic pattern can be best described as the wall, and this is just a random example of how fast playbacks I started on Sunday night during the game of Thrones season. Premiere season, around 6 p.m. known as the price as a prime time and I. Think at this point in time can be best represented by this image in terms of what API source is faced with, as well as the emotional state of our engineers and.

A

So looking at Game of Thrones, a traffic pattern episode after episode season after season left us with variants, only feelings about a future. Can we hold on during the next episode? What about next season and the answers were less than optimistic because we were running into multiple problems. Chief of them would be under utilization.

A

Rani Nagi implies that you can utilize a single CPU core at most now, since we're deploying in this ec2 instances, PC traces with single core, typically almost always do not have good Network. So to find good balance between network and CPU, we had to select instances which run at least two cores of CPUs. So right there would not be utilizing 50% of ICP use across all deployments under been scaling is good. However, it's slow or slower in comparison, and sometimes it is inadequate or slow to react to spike traffic.

A

Thus, we have to over-provision our deployments double and triple and sometimes more to accommodate for unpredicted traffic patterns. So you take initial 50% under utilization buffer up with over scaling and multiply by a number of regions that will be a lot of unused CPUs.

A

So as 4lb goes that every service communication requires a lb and again it's tried and true lb + hg approach. However, even for internal communication, but in same EPC will required stand-up lb for services again. That resulted in lot of lbs, which led us to problems, limits or otherwise known things that we run and out of it's ironic, because we were underutilized in more than 50% of CPUs and yet were running out of all other resources.

A

So to keep up with the resource usage, we have dedicated alerts which will fire up every time we cross 80% Thresh utilization on given resources, lbsg security groups and when we get modified, will contact AWS support to increase our quota limits. Of course, things got more interesting when we began to run out of instances for given instance, types or IP addresses for given subnets well, even a Tobias could not help us with our problems.

A

Then, of course, there are external resources that directly tied to each discount like a telemetry provider who would calculate the license usage based on the instances.

A

So this brings us to kubernetes, and rather than going through, these bullet points. They're, all true I will tell you my personal story. In fact, this is my second time being on the stage at the cube town.

A

First time was in San Francisco in 2015, when I was summoned to stage just like today by Kelsey Hightower, but my great surprise in horror, because I was not supposed to present what happened that walking through the hallways of cube, Con 2015 I dropped my wallet some way and someone have found it and returned it to event organizers, and he called me to stage to return my wallet so I cannot simplify in a better example of kubernetes saving the day. Many stories up and running and prevented and outages otherwise would be very interesting flight home.

A

So we I was telling communities from the server, but we did due diligence will look at messes. With this year's. We look at swarm. We look at the ECR and for us, police wasn't still he's a clear winner.

A

We were just beginning of a journey and we had a very tight schedule, given that we had to consider Izola services first and continues. Ation and mass-scale is a huge undertaking on its own, so once we begin country in that hill, so we start playing with Publius and AWS and again this is the end of 2015, a lot of change since then. So what we started with, we would most basic setup available run in Cuba to deploy kubernetes into existing pcs. We had to tweak and make some configuration changes and what we needed to do.

A

We need to show to appear store bosses to ourselves that kubernetes note is vaporware, but, more importantly, that kubernetes can be operated in a diverse cluster to production, great services and once we get, we started cutting achieve with Jenkins infrastructure cluster and once we've gotten successful, that we begin to provision a home cluster to a streaming services and that's when we realized that basic setup, simply one carrot and we had more work to do and Z will tell you how we got it done.

B

Okay, I'm gonna tell you something we learn from our kubernetes journey, so we create our own Tower from templates for our provisioning, our clusters, we started before some of the community projects started, for example, or Cuba, wsk, ops or cube spray. This allowed us to do something really cool. For example, we can deploy our clusters into our existing AWS infrastructure by providing our VPC ids subnet IDs, a security group IDs.

B

We also had high availability in mind. So from the very beginning, our minions and master aSG's are multi easy. The purpose of the SDS are different, though so for masters. We want to maintain a fixed number of nodes. So if one fails a the base will automatically launch a new master for minions. We want to scale up and down very fast, so we use ASG to launch a determinate nodes.

B

Master is also running in a chain mode, meaning that API servers are low, balanced and schedulers and queue controller managers are running as leader and followers. Despite be homegrown, we keep incorporating best practice from the community. We turned our oh I, DC or open ID connect authentication for a coupie API server. So as long as our developers, github accounts are in HP organization, they will get a token for their cube. Cut' or authentication tariffs or modules is a great way to promote reusability and modularity.

B

We create we created self-contained tariffs or modules for both communities, masters and minions. When we want to launch a new cluster, what we do is we compose a terraform template like this, and we will run terraform apply and bam. We have a cluster up and running. Several weeks later we have had some experience of how to operate a cluster, and then we noticed a few problems.

B

So first we run Prometheus II in the cluster to squirt metrics, with a provision I, ops, EBS volume as data storage, because our cluster scale up and down all the time. Sometimes the minion that hosts the privileges part get terminated. We have to wait a long time for premises, part to come back because kubernetes has to detach and reattach the EBS volume. The process could be very slow, and during that time we were losing metrics. The second problem is for big events like Game of Thrones, premiere or finale, or simply our regular load testing.

B

We have to scale pre scale, our cluster up significantly to overcome the war effect Illya just mentioned, and sometimes a de Bresse cannot provide it sufficient capacity of our desired instance type for the minions. So these issues led us to an improved version of our tariffs or modules. First, we added instance type arrival to our minion module so that all the minions launched from this SG will get this particular instance type. We also added a taint part to the minion module and passed that as a couplet start up flag.

B

So if this VAR is defined and all the other minions launch from this, this SG will register with that particular taint.

B

Again, we benefit from the modularity of our entire form code. So for regular minions, we pass our maintenance type to the 2d module.

B

We added another module to our cluster. We call the backup minion. These minions are exactly the same as or regular minions, except that they run on our backup instance type, which is C for the attacks, another module. We added with what we call reserve the minions so reserve. The minions are again exactly the same as regular minions, except that they are tainted by this taint to reserve equals true. At the same time, we updated our cluster autoscaler so that, when clutter scales down the mean our reserve, the minions are excluded.

B

So, to summarize, we add a two new mini aSG's to ours cluster. To address the issues we had earlier. First, if a device runs out of the maintance type, we want with killed about backup minion to bring more capacity a second for prometheus. We update the premises deployment to have affinity to reserve the minions and also tolerate the reserved taint. So in this way, Prometheus part is not interrupted. Even when cluster skills down flannel was the networking layer we chose at the beginning compared to other solutions.

B

It was simple to set up, especially before the CNI King, and it is included in every Korres distro, the distro that we'd use. However, when we were doing our regular load test, we discovered that on the heavy load there were some problems. First, it was increased latency and timeouts, both between parts and going out of the cluster a second there was UDP packet drop, which impacted our cube, DNS, lookups and custom metric collection, both of which are UDP traffic.

B

These are just two issues that we saw very frequently during the season to get up. Links are on a slide now. Let's talk about different types of services that we tried and the different ways we used to get traffic into our cluster. First, we provision ERP, is for every node port type of service and the associated easy OBS with the minion aSG's. So in this way, all the minions launched from the SGS will be registered with the obvious automatically.

B

However, there's a hard limit of 50 gr be spurious tree, and also since we are provisional, Yogi's ourselves, you have to keep track of them. Manually next is ingress, which I think is the most common set up out there. So we put a shared yogi front of ingress controllers and they'll be forward traffic to ingress controllers, which then proxies traffic to upstream services. However, there were some problems with that too.

B

First, when we looked at the cloud watch metrics for shared ELB and we noticed some 500 errors but which back-end services are serviced exactly, these 500 comes from its pretty hard to tell without scrunching the ingress controller logs or EOB access locks. A second, we notice, the ingress controller seems to be seems to struggle against a very burst or spiky traffic that we saw and this this tests, the setup produced more connection, timeout errors in our load tests versus the no port setup on the previous slide.

B

And, finally, the publicity of your shared ingress Lobby will determine all the publicity of all the services a last but not least, we tried no balancer service type, which is cloud provider. Specific dienes in this scenario, kubernetes actually handles the provisioning of the yellow bees and registered a minions with the Yogi's. This method is not affected by 50lb limit. At first, we noticed there was a API in the West API throttling problem a second. There was some ELB security group, customization issue that didn't get resolved until recent 1.8 release.

B

So- and this is these- are our choices for services and ingress for production services. We use no port plus individually Obie's for non production services. We use ingress controllers and share the Yogi's. In both scenarios. We use the built-in service discovery for making calls between our micro services.

B

Cube. Dns is always an interesting topic. Have you ever looked at the result? Account file in your parts. Basically, this is how it looks like first, you got a bunch of internal domain names to search for an author and a de brest domain internal domain names. So this code is actually from a part in default namespace. So you see the default or service to cross our logo there I'm. Second, you got your service IP of cube, DNS and finally, there's and those five options.

B

So this option basically means that there will be many invalid and not unnecessary, DNS lookups.

B

Basically, this is what happens so, for example, we, if you want to resolve a DNS called PG SQL back end I fell calm because it has less than $5 in it. We append all search domains and try them first before an actual DNS query happens. So why did and us fire was chosen, was explained in detail in this ticket by Tim Harkin, for reasons like same namespace, lookups, cross, namespace, lookups and, of course, crush the Federation in the next foreign slide. We will share some of the tunings. We have done to reduce those environment lookups.

B

Okay, these are some of the tunings that we found very important to us. First is the cache size of DNS mask container I? Think the default is somewhere around 100 or 200, but you should set it to max which 10,000, unless memory is a really constraint in your system, well set in to 10,000 all in cost. You addition to a couple megabytes of memory.

B

Address flag: it is a really big performance booster, so this flag tells DNS mask to return an IP address for a specified to my name. However, we use it slightly differently and we specify a whole bunch of invalid domain names that were created by it and those five, and we do not specify a IP address so effectively for these invalid domain name, lookups could DNS mask, will return not found immediately instead of doing an actual look up this way, we speed up stings a lot.

B

So if you haven't taken a look at your Kube DNS deployment, I recommend this flag. And finally, if you have some internal name name servers, you want to hook up the server flag is for you all. Your powers will be able to resolve internal domain names without additional changes, and with that I'll hand it back to Ilya thanks.

A

Quick word about telemetry: it's not surprising that we didn't have any continuation services before we couldn't take almost anything from our telemetry stock to companies except Splunk, and it was a Splunk telemetry team did some heavy customization and tuning for Splunk 40 to get reliable logs. Everything else on this slide is was new technology to us, and I think it was a great thing: zero dimension special case for reserved instances for staple service like prometheus.

A

We also love prometheus, however, running EBS with the availability zones and not affinity for Prometheus juggling of it. It's not fun at all and speaking of PBS. It can have some interesting month and a mountain times, so we related route with the great success and we just didn't risk to put in production before game affluent season. However, we excited to see you become an ciencia project. It's been submitted to TOC, okay, so foresee advisor is one thing to consume.

A

Metrics from see advisor run a new infrastructure cluster with few notes to CPU cores each and couple Jake is pod deployed it's totally in order to run on the 300 nor cluster, with 40 cores each and more than twenty thousand containers deployed consuming metrics at that scale felt like drinking out of the firehose, and we had to do some extensive metric tuning filtering and permit his memory adjustments to get metrics in reliable state.

A

So when do you know, you're already ready for Game of Thrones season premiere for us hit ball down by seven up the bar and in terms of threshold viewership and ramped up speed and beating it with the load tests? So for about two or three months leading to Game of Thrones premiere season premiere, we ran a weekly mega load tests and first attempts were just beautiful. It loves us in ruins and that's when the real work began.

A

It began on both fronts on services, side, a service engineer, did some heroic job investigating issues and fine-tuning services to accommodate for new environments and a granade aside. When we begin to look for issues reporting, if none were found and fix in what we could slowly, we began to purge in better shape, gaining more confidence in kubernetes and sources running on it. If there is any moral tale story after successful Game of Thrones, Season, 7 and kubernetes, it feels it feels good.

A

It felt good to be right, perhaps for the first time by making the right choice, and if there is any advice we can give is trust yourself, trust your team succeed. The small things and you'll be well positioned to succeed at big initiatives, and it won't always be a smooth ride, but you and your systems will emerge in better shape than you went in for us. Many problems we found in a services were not caused by kubernetes that were there all along kubernetes made them more visible.

A

So, as mentioned earlier, we looked in alternatives, however, the biggest and in the undeniably most important reason why we chose kubernetes, was vibrant, inactive kubernetes community without all the github issues and fixes, without C groups and slack channels without meetups and coupons. Just like this there's high chance that a journey would not end well and will most likely end up like these two guys, but likely didn't happen, and here we are at Cube, can't tell on a success story of Game of Thrones in seven and communities. Thank you.