Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Governance on K8s: How to Solve Ownership, Metering & Capacity Plan... Micheal Benedict & Yongwen Xu

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Governance on K8s: How to Solve Ownership, Metering & Capacity Planning - Micheal Benedict & Yongwen Xu, Pinterest

Pinterest is a cloud first visual discovery engine that serves over 250MM users. To support this scale, there are thousands of services running on tens of thousands of hosts, processing 300+PB of data. We operate large kubernetes clusters across several availability zones, across regions. The cluster is auto scaled with support for pod level auto-scaling. Finally,to effectively utilize resources within the clusters, we operate heterogeneous workloads on a kitchen sink of instance types. Given this, 1.Who owns what? 2.What is driving utilization? 3.How do we plan capacity effectively with minimal overhead? In this talk, we will share how we built a governance platform to address the above through defining canonical ownership, metering resource utilization (at various granularities) + reporting and finally a policy enforcement mechanism (ex, pre-emption, placement, etc).

https://sched.co/UaYq

A

All right, hello, everyone thank you for actually staying for the last talk of the day. I know everyone's gonna go back home after this right. It's all exciting, so we'll try to wrap this up really quick. So we can also head out cool, so my name is Michael Benedict I'm, an engineering manager at Pinterest I lead the engineering productivity team who focuses to improve experiences from code, build deploys, and obviously what we're going to be talking about today is more about governance and I.

A

Have here with me, young men, who's the tech lead of the team and we're here to share some great stories with you on how we helped improve our build a governance platform at Pinterest, so to get started. I'll go through my usual slides first off. If you don't know about Pinterest okay, it sounds like many of you won't I just quickly share with what we do. Our job is to provide and provide everyone with an inspiration to create a life they love.

A

So Pinterest is all about creating unique experiences based on your interests and providing a path to actually pursue those interests as well. For example, recently I ended up cooking steak, I lived up some recipes at Pinterest and it really worked out well.

A

In fact, some of those recipes were so awesome that I ended up buying the raw materials from Pinterest as well, which helped me create an amazing steak, but regardless our job is to help you take some inspiration and actually execute on that to help create a life you love and obviously the showcase why this is such an important thing. We have tons of people who use Pinterest. We have over 300 million people who use Pinterest on a month over month basis they have collectively created and curated over 200 billion pins.

A

That's like the largest curation of pins or assets across the world and they're all been categorized across four plus billion boards, and our job is to kind of take that and help every engineer at pin just move extremely fast, so our job is to provide a fast, safe and a delightful path from an idea to production without having anyone to worry about infrastructure. That's our job.

A

So what makes this really hard the pinterst architecture itself. So pinterest is not a simple system I like to quote our chief architect, who likes to think of this as a left and the right brain where one 1/2 is like dedicated to analytics, where we take in so many different inputs and try it on this and how we can help create the right.

A

Inspiration for different sets of people and the other half is more around the serving stack itself, which takes a lot of this content and data and help serve that to tens and millions of users. You can clearly see at the top of the stack on the left hand, side. Those are our clients who use the Pinterest app both on the web and the mobile, and it's all powered by a large API service and for quick context.

A

Those two are the largest services at Pinterest monolithic, Django applications which exist even today after 11 years after its first commit, it's been hard to kind of get rid of them. So with that, you can also see that the layer below that we have a tons of Micra services which actually provide the core business logic for Pinterest and all the way at the bottom. It's more or less. Our persistent store stack all right. Obviously, all that infrastructure as complicated as it is.

A

We have a huge footprint so overall, like thousands of micro services, where we do over tens and thousands and deploys a day. These run are now getting hundreds of thousands of servers, and the batch system is solved. Powers even more like millions of workloads running almost on a monthly basis and what's more interesting, is obviously given the size of infrastructure. Our costs are not small or low.

A

So to give you a sense of how we actually divvy up how much we're using on the public cloud, this kind of gives you a quick overview of how our costs are actually divvied up. So for some context, Pinterest started on the public cloud from day one our founders, the first commit they put together for the Django application, was deployed on AWS ec2 and ever since then, we've always been on the cloud. So we are probably the cloud native company you could talk about, so this is kind of how our costs are actually divvied up.

A

And what I'm here to talk about today is if you're running on the cloud. You probably know that it's not as simple as trying to figure out where your money is really being spent and what adds to this complexity is when you start looking at each aspect of the cloud. So, for example, we rely heavily on the primitive services which are public cloud provider, offers the compute infrastructure instances more or less the storage infrastructure which in this case we're a big AWS fans.

A

We run on s3 and finally, there's some other costs as well, which you can talk about more. But what really makes it complex if you start to divvy up our compute costs? It starts all the way at the more foundational primitives like regions and then availability zones and then gets even more complex as you go up like okay. What are these reserved instances? What are these on-demand instances?

A

Spot instance mean, and there is these tens and thousands of different instance types which we have to grok and understand, and they all mean something, and it will even complicate matters worse.

A

Everything can be put together in concepts called auto-scaling groups where it can scale up or down, based on how you've configured your resources now imagine authors makes it extremely complicated to understand how you're spending your money across so many different dimensions and, to be honest, we're not done yet, even after that, you actually end up paying more money, as you kind of transfer data in and out of your instances to different variety of services.

A

So this slide really just shows you a complex view of sure you're spending, money and compute, but in reality it's going through all these different aspects or dimensions of your compute print.

A

It's to give you an overview, the same applies for even storage storage is not easy. Just storing a file on s3 or any like GCS is not the same, because you have to pay again based on the dimensions of what you're trying to do. It could be stored in a particular region or multi region, you're, probably storing all the standard buckets or it could be on glacier and any other aspects or tiers of offerings from the cloud priors.

A

So overall I mean it's it's hard. It's really hard to kind of drag and put all these numbers together and truly understand like wearing exactly the money is going and that's the biggest challenge we had at Pinterest. We are so large that it became truly hard to understand three things who is running? What on the infrastructure? How does it all add up to a certain amount of cost, our set of resources? And, finally, what can we do to better understand a plan going forward for all these different resources? I Barney?

A

That was a gift by the way, so alright, so infrastructure governance, that's our seal of approval, basically to say that if you're saving money in our company, you should get that seal of approval, because we don't want people to be sending email saying that I've saved tens of millions of dollars without actually reading it up. So that is, you know, props to my team to kind of put that together.

A

So I'm gonna cover like four specific topics along with the onlin over here. So how do we build a platform of governance at Pinterest? There are four key pillars as we like to call it: one: visibility: getting insights into how and who's using what to ownership. This is one of the most important things we'll spend some time talking about. How do you have a clear view of ownership of who owns what all the way from resources to code to all different aspects of tools and systems? The third one being?

A

How do you effectively plan budget and capacity? And finally, we'll just briefly discuss a few things on the optimization side, specific to Cuba Nettie's on how we're improving that front cool? So, let's start with visibility: what is visibility so there's gonna be three aspects to visibility and I'm sure everyone's probably dealt with this if you're running on the public cloud there's provisioning. So how do you provision resources? You can use your API or different sets of tools like cops. Are your own custom deployment or cluster management tool as well?

A

The second thing is: how do you inventory these resources, so you have provisions some resources in the cloud. Great, do you know actually where they exist, and how can you view all of them I'm sure that consoles are do a good job in providing that data, but after you reach a certain size, a size and scale, it's impossible to look at hundred thousand server in the same console console page and finally, how do you understand the state of these resources? Is it running right now? Has it stopped? Has it been killed?

A

So there's a lot of content or information which is missing, which usually the consoles don't provide at a certain size and scale. So those are the three key things. It's part of getting visibility. How are you provisioning? Your resources? Can you invent your resource and attract metadata associated with it? And finally, how do you meter and understand the state of those resources as well and resources over here I'm using it pretty loosely, it could be instances, it could be buckets, it could be even pods and with that I want to like do a quick plug.

A

We had open source in inventory, store at Pinterest. This is what we use. It's called Soundwave externally internally, we call it CMDB configuration management data store, not a very friendly name, but yes, this is what we use today, where we can query every instance running at Pinterest. We can get very precise metadata about when it was terminated when it was running like what was the configuration around it all the way from network cards attached to it to what kind of versions of OS we were running on it.

A

So this has been an amazing store for us to keep track of things as and when instances come and go and clearly that lets us do things like this. We can truly understand okay, given a number of instances, and you have all this amazing metadata next to it. How can you truly visualize what's happening within your company? This is just a quick tableau visualize. You should be particularly just to show this footprint of all the service is running at Pinterest.

A

The blobs you see and the size of the blog reflects to how many instances and CPUs they are using essentially.

A

So this thing does get complex the moment you get into Cuba Nettie's, because when you're looking at instances you do have access from the cloud provider on the API and metadata to tell you, you know when an instance came up when it died and other associated metadata, not to replicate that similar experience. It was important to look at kubernetes and see what kind of primitives to behalf in in place to kind of leverage or create the same level of metadata. So we can reuse our same system.

A

So some of the key things we realized is that this data is ephemeral. It's only stays for a certain amount of time within the Cuban eighties cluster, so that was one of the biggest challenges we had to address like to ship those information that information as quickly as possible to some sort of persistent store, and we ended up like leveraging elasticsearch for that to ship this to our inventory store and that's kind of a key metadata. We try to like pull out from every pot like what is the identifier when was it created?

A

When was it terminated and we also track all the reservation information in terms of the resources requested by that particular pod and all of that gets shipped to Soundwave. One thing: you'll note it interesting lis. There is a concept called identifier now these are not pod names and very soon, we'll actually talk about why this is one of the most important things. If you really want to have a clear view of your infrastructure from instances to Kuban at ease, it could be even Hadoop for that matter.

A

How can you singularly try to understand how can to identify these resources, and once you have this data in Soundwave, it was more or less an ETL job, which we ended up. Writing called a chargeback system which pulled a lot of this data together and compiled information for us to tell us how resources are being utilized within the company, and what this lets us do is let it lets us visualize, this particular data in very different sets of dimensions. This could be from how many instances are running to up say who owns a particular instance.

A

Even more data saying group, even more data, around aggregation of which team owned certain instances of which organization owns instances and what's the best part, is since all of those been generalized, we can actually look up a given team or a given organization and only see the instances they run, but also their Kuban at least pause. The s3 footprint, the Hadoop just they're running all different platforms, all bundled up through a single chargeback interface.

A

And yes, it leads to some fancy visualizations. This is kind of the data, our fin ops, team and others track on almost a day-to-day basis to see how we're going on CPUs. This is literally just I think four months worth of data, but you can clearly see the uptick in in our V CPUs. That's the metric. We use in two just to show how we're prepping up for the holiday season.

A

So, yes, all of them, each individual color actually represents a service or a platform and I think it should be pretty obvious, like the large ones. What what they essentially mean. Why is it hard to kind of track the large platforms?

A

So one thing I? Do want to call out is that this is just an ec2 or an instance view of things. The next level of charge back actually gets you details into. Why is that blue bar so large, so that is potentially either Hadoop or cubed Ares in this context, and we can actually do a deep dive to understand which pods are who's using what resources within that as well all right. So with that I'll actually have young men talk about our next pillar, which is identity and resource.

B

Thank you, Michael hi, everyone. My name is a young racer from Pinterest, so I continually talk about the other expect of the invoker by Pinterest, so ownership as Michael mentioned. That is the cornerstone of the covenants, so it we need to know that who owns what we sauce and then how the ownership is go up to which organization and then who spends what the kind of things and they also. Actually, we want to know that what manager owns this resource, like this project right, that's very important things to do.

B

Okay to solve the ownership problem, we build our identity service called Timbers, so we don't have name here, but its consumers and it had three components: one is identity and the ownership and the metadata so itself as a single source of choose that the ownership and associated metadata.

B

So here's a so there's a there's, an example of the service on uncommon economics that we talked about in yesterday. So it's a. We have the ownership of this service at the the Nimbus system, so it shows what the owner is as a manager Michael and also at the teen youth team, email and then description also there's some metadata associated with this service. So, for example, we can easily find a threat information.

B

You can find a piece of duty of this service so that asahi can easily two pages per page, the owner of the services or anything that goes wrong with the service way yeah.

B

So, let's take a quick look on how the how the identity was specified in company the world, so we implement this as the key identity. The key to that ended here is the canonical specification. So that's why we have this URI. We call its call unique resource identifier. So that's we have a specification. That's always start with numbers, just like any HTTP, so these numbers and then you can put the project there and then the platform and then each platform. We can have a small, slightly different specification, for example.

B

Here then we have the company the platform and the vacuum uniform. We can identify the cluster. This part is landing ad and also the what go type so App inches. We use order, Ciotti and then the CNT type. Actually we started in here and then also the name of the application, the pod, so in this case that actually this information is stored in a notation that is injected by the or CRT controller. That CRT controller will depend on the namespace and then that's this.

B

These services belong to what project right and injects in notation in, in a part so that later on, in the data injections, the ETL pipeline were able to identify this part, and then it's a utilization of this part as well.

B

Okay, one thing to mention is actually use this URI format so that we can support multiple platform that we going to a certain project. For example, then this project, this this service, can also run on vm, like the ec2 environment, so that you can have another another resource that is called Nimbus and then same project, my tab and then on ec2 and then Ann Kuster and their stage as well.

B

There's this way that we have global view of a project that what we sauce is beyond tall and also we can add s3, for example, or even if this project associate some Hadoop job, then we also group them into the same project. We have a different platform, different resource.

B

Ok, with this ownership system, then we can actually build tools along that beautiful longest. The Nimbus service right the tools. One tools we talk about here is the asahi part. Hi, sorry about is actually a select part so that you can interact. You can run a command to do what we were clearly on project or even on a host name on the pot name. So from the host name of her name, you can.

B

We can be worse clearly that what project is parties belong to and then from the project we know who, which manager actually own this we saw sway and then, if anything that goes wrong on the underserviced and you totally what's woke up, then we can find a page of duty and then you can page the person that that's on call for this service. So it's very useful as it's honest, your true okay. Now now we have the visibility and then we have the ownership identity.

B

The next step is to look at the capacity planning and because we have the history data by the capacity data we can do forecast and then the grey line is the great one is from the the planning. Basically like. We can look at the previous trend and then we do some kind of folk and then the red line is the actual usage. So you can see that as some small variation there and the goal is to keep the variation less than 5%.

B

So that's that's the main goal of the capacity planning which might not be that easy, but that's the goal you want to do and we're going to do a case study here. So even we plan really effectively and then things still can go wrong in this case right. This is this a crystal study that we actually have a sample one instance, so one of our major service that it's this building across a multiple failure domain.

B

It recall, availability, zoom, easy and then there are other scale across three AZ, but in this case you can see the line there. That's a one. Each line will sponsor for one AZ right and then you can see that one line actually is for people. So that means actually this easy one, not of capacity. So we have that easy.

B

We were able that we only the copper wire that is not able to scale up the machine for us, so that indicates that we cause the CBO bending and then actually overall, the one of the atoms stream service and cost us a job. The solution for that happening is to increase the HEPA, depending so that we can have able to allocate enough capacity for all the AZ right. So that's the one use case study that shows that capacity pending is very important.

B

Okay, now that we have a feasibility, we have ownership and then we are planning. The next step is to actually do something right with the action part. How do we proactively save money to energy, achieve lower efficiency and then optimization go so I'm going to talk about three actions that we did to achieve the goal so from the chargeback system. That Michael mentioned that we have visibility of the different things right, including the instance type, so instance type.

B

You know that different engines have a different cost and then we did a project that my great from c3 instance type to c5 instance time. We only so that's that it away, as instance that way, but in this case, that the migration actually took a long time to complete, because all the different service owner have to migrate it manually and then they do is separated manually way.

B

And then you involve third study past him because we are at the beginning with all your Syria type and then c5 time is much efficient and then ganja and then a cheaper right. So that's the good thing about c5 and then that's why we might take a long, a long time of other resource, but we did achieve very good efficacy game and in the future that we we are migrating. We are migrating, I mean the service angel cake and Cuban. A turkey would include near the water.

B

We don't really need to worry about from so its own Apollo. We don't really need to worry about the instance type, so this will make the migration much easier to say in the case that we need to move to other instance type to save money, then we have culminated and the things will be much easier way. So that's a good thing. Another thing is another example is to look at the spot instance as well.

B

So as we know that spot instances it's much cheaper than on-demand instance and then for us we, we first identify the workload that is suitable to run on the spot instance, in that case that we can migrate them and that reduces the cost of those servers. So the second action we did is to maximize the auto scaling. So this is a graph that shows the number of parts of a service, so it auto scale according to the load. In this case we specify by the CPU metrics.

B

So we utilize the Kookaburra HPA, the horizontal autoscaler. So we define the HP a policy that if the cpu is above 50% that we scale up and it's big old and we scale down so that achieved this file, export difference during the pig and in a downtime and then there's another scalability is on the Noda level. So we use our cursor autoscaler. So it's also open source CA that this they say actually the other scale under on the node.

B

So whenever a part is pending on the scheduling and then we there's no no no to available and the other scaler kicking, we actually have. We actually implement a buffer way. So that is not actual wait for not able to schedule them. It's a scale up. We have a buffer so that we can scale up earlier than actually did so. One thing we want to mention is auto. Scaling is a good way to achieve efficiency way, but it also posed a challenge for planning right.

B

If you can see in the past so I that, let me see you can look at the pass right so in this guide, and so that is the graph for the service right. You can see that we have a pattern that on the doing actually doing whatever five days of a pig and then during the weekend, so the utilization is much lower. So this pattern is pretty clear, but if you look at the node, so that's the test on the node side. So the no sided, that's not actually curly have these patterns.

B

The reason for that is because we are using the we are using kubernetes share platform echinacea perform. We are multi mixed workflow on the same platform, so different part. Different servers actually have a different scaling patterns away, so when we do it being packing on the KS, so the the pattern is not clear, so this cost actually make the capacity planning much harder.

B

So we don't really not really not able to predict what kind of usage pattern that from different service and so there's something that we are going to go, get our data and see what's the best model to do the prediction: okay, next things there's actually their third action, every Jewish to utilize the quota management.

B

So as we know that KS have the built-in quota management to make, the big I mean the whole time and gestures by namespace way and then because we have project mapping mantle one to the identity, metro namespace, so we can easily to control that what project the quota for each project right. So this is an example that you can do on the actual ml file that you can specify on the communal platform and we can specify how many cpu we can advocate on this namespace.

B

So, for example, we have some development books, workload away, you can put in a namespace, so that is 201 there and then, if there's one of those another capacity, then it's not the SOA for that service world. Oh, it's not as important as the production workflow. So there's something that the coder manager helped us a lot. Okay.

B

So, with that hand back to Michael to have a conclusion to the talk, thank you.

A

I think they're working okay, perfect, so.

A

Can you still hear me okay, so two things I want to call out is before I get into the takeaways I think when we move to Cuban Aires. One of the first things we realized is everything is auto-scaling at this point. There is no way we could be like running multiple clusters and per use case, because operationally that's just much harder for us, so we made a decision collectively to have a shared cluster.

A

The intent with that was to make sure that, as we are scheduling workloads on it, we have a more heterogeneous collection of workloads running on it, and we can focus more on packing them more effectively. At the same time, leaving enough Headroom for the Kuban, a nice cluster to grow or scale down, we've had couple of incidents in the past where, if they keep it too tightly, we have had run into risks where pause would want to be scheduled and the clusters just don't scale up more effectively.

A

The reason we showed that said zero is that, no matter how much of our planning you do, there is no way you can always have infinite resources from your cloud provider, there's always some gap or the other, and that's something you should always plan for, and so that was like one of the key learnings we had as part of that with that, like I, think the larger things for us was that it was very important.

A

None of this would have been possible if we didn't have a good way of understanding how to tie ownership or identity to different aspects of our entire cloud footprint. It was very important to kind of group all of those things, regardless of ec2 instances or cube entities. Pods get a clearer view of how it's actually distributed across the cloud footprint and part of that is only possible because we actually invested over two plus years and getting that identity system integrated with multiple platforms within the company today.

A

That has enabled us to like query and a group and do certain parts of planning simply because we have a canonical way of identifying anything and everything running on the cloud footprint literally. There cannot be any instance or pod running or even a bucket filled without that particular unique identifier in place. So this is a must I mean this is like the biggest learning we could recommend and I know. People have liked use or try to do this in different aspects by using tags and other features which the cloud providers provide.

A

I'll be honest, like none of them actually work. We have tried that, and it's just over time they get pretty stale names change and all of a sudden you have to write regex is to truly understand which clusters make up who, and that is something we just decided. It's not worth it so investing time into defining aspect.

A

How can you like model your cloud in a way where you understand what region means or what a cell or a means, and how do you want to like fan out from there from clusters to apps to pods was extremely beneficial for us, and we think that is something we can never er definitely contribute back to the community as a common spec to identify resources within the cloud.

A

The second thing is, we realized that it was literally not a tools problem. It is literally a cultural problem, no matter how much of our tools we build if those systems were not integrated and if the engineers themselves really did not adopt a lot of these best practices. There's no way we can get better visibility in, don't know who's running what in our infrastructure, unfortunately, we have like so much of data.

A

I cannot really show how much on the slide, but today, literally from our CFO, to an engineering manager to anyone within the company can drill up and down into understanding our cloud footprint in all aspects of resources and everything, and this is only possible because it was important for us to educate what these things meant in a very friendly and in a very user friendly manner.

A

And this involved us again like multiple years of investment trying to like educate about the chargeback system or about what identity actually means within the company and then subsequently generating reports. Notifying people saying that you've used XYZ resources and investing into a single pane of glass, new Y as well, where, if you're starting a new project at Pinterest today, you'll actually go through that UI.

A

First, which will ask you a bunch of questions before your entire project is actually set up and as part of that, all the identifiers are set up and they are like distributed across our entire platform. So clearly, that was a huge change for people in the who would just use the CLI and spin up thousands of instances. And yes, I'm sure man, it must have happened for multiple people here and yes, we have spent hundred thousand dollars or more just by you know, losing or getting into that run.

A

So this is a big thing and these things do take time, but when you build the right tools and systems- and you have the right guard rails in place, I think we can really make a difference in terms of and moving into the space I think with that. Finally, I would just like to say thank you. Thank you for spending the time with us today and yes, if you do have any questions, we're happy to answer offline and even show some other information we have but yeah thanks a lot.