Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2016 (Seattle), 16 Nov 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk - Micro-Services Lifecycle Management at Twitter by Micheal Benedict, Twitter

Description

Lightning Talk - Micro-Services Lifecycle Management at Twitter - Micheal Benedict, Twitter

"Twitter and its associated products are powered by thousands of micro services. Managing lifecycle of services (i.e create, provision resources, deploy, meter, charge & deprecate) at our scale proves to be challenging.

In this lightening talk, I'll give a quick overview on the problem & how our internal product (called Kite) is helping Twitter manage the lifecycle of creating, deploying and operating micro services at scale."

About Micheal Benedict
Micheal Benedict leads Product Management for Twitter's Cloud Platform. He and his team are building the next generation compute infrastructure that spans private and public clouds. Previously, he was a Software Engineer at Twitter building Observability. Before joining Twitter, Micheal was a Research Assistant at University at Buffalo’s Blue Labs and built PhoneLab: World’s Largest Smartphone Experimentation Test Bed. Micheal has a M.S in Computer Science from University at Buffalo.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hey folks, my name is Michael I'll be talking about the challenges of managing microservices at scale, so it's really hard to condense, like quarters worth of effort in like about five minutes. So please do reach out to me at Michael. After the presentation for any questions, I won't be deep, diving into implementation, but we'll be sharing some of the learnings from managing thousand-plus services at Twitter.

A

It must be not a surprise, but that Twitter used to run the largest Ruby on Rails installation for the longest time it was aptly called the monorail, and, given that routing presentation, the business logic itself were tightly coupled in a single place. It became extremely difficult to scale debug and work on the monorail, this kind of led to the decomposition of the monorail into individual services. Today, Twitter is actually powered by over thousand plus services and this particular chart, which is one of my to go charts.

A

What you're looking at is the specific request being served from our reverse proxy called the TFE, and this particular graph shows that create a tweet request, the TFE routes, all the requests done to the individual different services.

A

What's interesting about this is so moving to the micro. Service has enabled us to do with the obvious things right with the assumptions about ownership, enabling fault, tolerance and graceful degradation as a first-class feature for services, teams, owning and planning and scaling the individual services.

A

All of this subsequently improved the developer productivity and also enabled us to actually scale up our engineering teams without stepping foot on anyone else. However, what many people don't know and what we've realized over time is that there are certain challenges which are not so obvious.

A

For instance, the service oriented architecture did provide us a certain level of comfort around fencing and ownership. However, it is only valid when the organization stays intact. Not many organizations stay that way, there's always constant churn reorganizations and many things happening projects moving from one to B so many times what happens, services go unknown and, if not made visible at the right time, there is a significant there's, a significant risk of knowledge being lost and a potential impact both on the business from an Operations perspective, as well as the efficiency perspective.

A

The second challenge is about managing metadata. In general, our SRA and the ops team really appreciate a single consolidated view of everything that is critical to run the business. This enables them to monitor, detect, manage incidents and recover effectively.

A

This, however, becomes a challenge when the organization actually grows from a few teams to about 100 plus teams, thus there's a need for a centralized service discovery, service directory and a metadata management management system where you can store important metrics such as tier information, you're, monitoring, dashboards, SL, o--'s and different queries, where many developers from the organization can query and use them.

A

We also ran into one of the biggest problems, that is, the ability to identify a service using a canonical identifier, I'm sure many people deploying services onto public clouds or even hybrid strategies. Well, we'll be able to appreciate that problem, because not everyone uses the same name and their provisioning resources on these public clouds. We didn't have identified provisioning early on which led to every service having disparate identities across our infrastructure.

A

Usually organizations can map or consolidate all these identifiers through various programs which will last for a couple of weeks manual mappings of these IDs. Together, but unfortunately, this is, this is a significant tax on engineering.

A

Doing these cycles again and again adds to a lot of manual work which we wanted to avoid, and the worst part is by the time this manual work is done, there's a very good chance that all of those data you collected and mapped have gone out of date and, finally, the lack of consistent weight of provision and managing managed resources as well. It is very important to provision resources and tie to individual teams such that quotas, utilization, utilization metering, managing the various clients all become extremely simple and have to a single place.

A

This enables us to do some really fancy things to tell us. How much does the service cost to run? How much does a Team X spend on certain infrastructure resources kind of moving on so I think we can consolidate this into two big teams.

A

Essentially so platform operators, at least at Twitter, definitely need that tools to operate and manage their service for their customers and service owners themselves needed in an end-to-end lifecycle manager where they can go into single console view the projects and manage them respectively and that led to us building something called the kite service lifecycle manager.

A

So just quickly wrap this up. You can argue that we start off with the data center of the public cloud, where individual platform teams may run managed solutions or even use solutions which offered by the public cloud themselves. We build specific provider, API sand adaptors to enable these individual services which which solve the problem of identity management, single place to provision quota and do utilization, metrics and even charge users and teams within the company.

A

All of this, coupled with specific service life cycle workflows, which enable them to create build, deploy their services along with a whole different side of reporting engine, which shows how much does it cost to run? Your service all happening through a single dashboard and to just give you a particular example of what this may look like is. This is one of the chargeback bills we sent out to our tweet team.

A

Unfortunately, I cannot show any of the dollar figures, but really this is the level of detail we can get to, and in fact this can be drill down even further to really understand which particular applications, jobs and other things that running across our infrastructure. On that note, Thanks.