Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Autoscale your Kubernetes Workload with Prometheus - Frederic Branczyk, CoreOS

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Autoscale your Kubernetes Workload with Prometheus - Frederic Branczyk, CoreOS (Intermediate Skill Level)

Time to autoscale your cloud native deployments, but how do you make it happen? In the past, easier said than done. Lack of guidance and inconsistent implementations of solutions have made autoscaling on Kubernetes a pain. Tedious extensibility and difficult maintenance with Heapster were some of the causes for this. Those days are over! At Kubernetes sig-instrumentation, we have developed and standardised the resource and custom metrics APIs. These APIs are finally giving Kubernetes the autoscaling capabilities it so desperately needed. Frederic Branczyk, software engineer at CoreOS, will explain the history of autoscaling on Kubernetes, elaborate on the design and usage of these newly developed APIs, and describe how they benefit the consistency of autoscaling. He will talk about the recommended way to autoscale Kubernetes using Prometheus, and end with a demo showcasing just that.

About Frederic
Frederic is an engineer at CoreOS contributing to Prometheus and Kubernetes to build state of the art modern infrastructure and monitoring tools. He discovered his interest in monitoring tools and distributed systems in his previous jobs, where he used machine learning to detect anomalies indicating intrusion attempts. He also worked on projects involving secrets management for distributed applications to build sane and stable infrastructure.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hello, everyone I'm Frederick I work at Red Hat, which was if you haven't heard previously or recently acquired by Red Hat, know. Korres was a quite pirate hat yeah. So today, I want to talk about auto scaling your workload on on kubernetes with the metrics from Prometheus.

A

But what what do I want you to walk out of this room with in terms of information, so I want to want everyone to understand what is the officially recommended way to auto-scale workloads on kubernetes I want to walk you through the history of this stack and this architecture and how it has evolved over time to understand why we are where we are and then I want to give everyone a future outlook of where I know things are going and some predictions of where I think some things will go so without further ado.

A

Let's talk about auto scaling and first I want to make sure that we all understand the terminology and have a common understanding of what we speak of when we say auto scaling so on a very abstract sense. We want to be able to cover demand of our user facing applications, or maybe not user, facing really anything that what our customer is, whether that's actually an end user or some developer within our infrastructure. It doesn't really matter.

A

We want to be able to cover our demand based on our metrics and in order to be able to do that, we obviously need to collect those metrics, but really fundamentally, why we, why we do.

A

All of this is in order to cover our service level objectives, so things like latency of our AP is or the time within which we want to or to our users that we work off items in a work queue, for example, whether these are actually good at good examples depends on your individual problem, but that's generally what an SLO looks like of your service level agreement and, more often than not, we can't actually directly say that we want to auto scale directly on our objective, but we use something called a service level indicator which, as the name says, indicates that it influences our objective.

A

So an example of that could be in order to process some requests within 5 minutes or something our queue length needs to be worked off rapidly, so we want to make sure that our queue length is below 50 or something like that. It's again whether that actually makes sense. In your case, you need to evaluate that.

A

But you know your applications best, so you just like with other monitoring, need to figure out what are the requirements of your users and you need to set set those and collect those metrics in order to be able to know if you're performing the way that you're promising to perform. Otherwise, none of this really works and that's why we want to do auto scaling.

A

So in terms of auto scaling, there are two types of auto scaling and I will mostly be focusing on the first one, but I will still be explaining the differences, so there's horizontal, auto scaling and, more specifically in criminales. We call this the horizontal part, okay, autoscaler, often abbreviated as the HPA, and what this means is in order to cover our demand.

A

We increase the number of replicas of our application and I think this one is particularly important because, although there is also vertical pot, auto scaling or just in general, vertical auto scaling, meaning we don't increase the replicas of our application, but we increase the individual size of our replicas. So while vertical part, auto scaling is some is very important as well.

A

It becomes impractical to scale beyond a certain certain size, because, even though we have really giant servers out there with two terabytes of RAM and gigantic amounts of CPU, even those hit some physical limit at some point and become impractical to scale out. So what we often focus on is horizontally scaling because within cloud infrastructure on with our cloud providers, we can basically treat it as endless resources unless we're, maybe a Google who scales and data centers, not in instances, but even that, even on that kind of level we we could do.

A

We could apply the same principles and just to cover cover this again. We are demand. The necessary amount here is calculated in terms of kubernetes the resource requests and limits.

A

So now that we've covered auto scaling in abstract sense, let's look at how this was historically done on kubernetes and depending on the kubernetes version that you are running. You may already be using this stack and, as you can tell by my language, there is something new coming up or already there so historically, auto scaling was heavily based around a component called heap stir in kubernetes and heaps.

A

There is a monitoring system that collects metrics from your infrastructure, mostly from a component in your kubernetes cluster called C advisor, which sits, which is actually compiled into the couplet in kubernetes and through that it gathers and infrastructure metrics about your containers, things like CPU memory, usage and there's also some possibilities to do custom, metrics collection and then, when hipster goes around and collect all of these metrics.

A

It then writes it into some sink so into some other time series database and when we wanted to consume this in terms of horizontally scaling, what we did is we created a horizontal part, autoscaler object and how this object works is that we have a reference to the particular resource that we want to scale.

A

So the in this particular example, we have a deployment, but this could just as well be a stateful set or really anything in kubernetes that has replicas in it, and actually this is generic in terms of kubernetes, because anything that is scalable and we'll go into that. A bit more in detail later can be used here, yeah and then the next fields, I think are obvious.

A

So we always want to make sure that at least five replicas are available of this, and just in order to not explode the resources that this needs, we can also specify a maximum replicas, and within this design there was only the possibility to do auto scaling based on CPU usage and more specifically percentage, so the resource requests in terms of CPU that we give it if it has 70% in this case or more, then we start spinning up more parts of this deployment.

A

Now this is this was really cool. We got this in criminales 1.2 and we could auto scale and, in theory, all problems out. There can be converted to a CPU bound problem. Now you probably already understand some of the problems with this, which is its while it's theoretically possible, it's not very practical to do this, based on CPU usage. We may very well have something that is much more efficient when it's EEP memory bound or really it's very hard to boil down our SL O's to something like CPU percentage.

A

Some organizations may be able to do this and it's great for those organizations, but it's very hard to get to that point, to be able to do that and over time, as we developed hipster, so hipster only started with a few theta betas things, so we were super excited when all these vendors came in and said yeah. We want to do auto-scaling with and monitoring with kubernetes as well, and we said yeah all these vendors are behind this.

A

What we want to do this is really cool and, as it happens so often in open source projects- and this is perfectly normal- is that not everybody maintains the code that they contribute and I think this was less a problem of the community rather than us, the developers of hipster who made the architecture. This way, and so over time, this created a lot of problems. It caused outages and kubernetes clusters. Monitoring is basically the most important component of your of your infrastructure, so it needs to be extremely stable.

A

Google actually has I, don't know if they still do, but I know that they at least used to they have in terms of priority. They have something like medium high and monitoring. So monitoring is the most important schedule, scheduling priority. There are other organizations doing this, and so we want to.

A

We want to make sure that whatever we provide a super robust and what's also, what also was strange was because was that, because of how hipster works that it it collects, it must collect the metrics and write it into some time series database, the big the big monitoring system that is super popular within kubernetes, which is Prometheus, fundamentally, cannot work with this and then heaps are also exposed, metrics and Prometheus format, but we can't use Prometheus.

A

So this was confusing to all of us, so we we decided about a year and a half ago that this pipeline needed a redesign, and we wanted to solve all of these problems that we had previously created ourselves with heaps there, so that we can go forward and just be productive to auto-scaling and stable monitoring.

A

So one of those things that we wanted to achieve here is not be bound to only be able to scale based on CPU. We want to be able to auto scale on arbitrary metrics and what we came up with are were the resource and custom metrics api's within kubernetes, and these were specified about a year and a half ago and are actually available in all kubernetes clusters, starting kubernetes 1.8. So most likely all of your clusters already support these.

A

So, what's important to understand about these api's- and we made this very very much a goal of these api's is that they're just a specification of api's, so the individual implementations are up to the vendor to do so. We don't have this model again, where there's one repository where everybody puts their integrations and we again get unstable and to end tests in kubernetes and in general it slows down the entire kubernetes development.

A

Instead, these vendors or the community around these vendors or other crew open source projects need to write these maintain these and maybe hand off ownership. So this is it's more explicit than whenever anyone is using, creates one of these adapters that they're responsible.

A

So that was important for us and something that's important to understand when using these these api is only return, a single value, so the most recent value of that metric that you're requesting so the resource metrics api is actually very well defined about the metrics that it does expose. So these are something that we sometimes also refer to as the core metrics, so things that every workload in your kubernetes cluster has so CP.

A

You memory file system, those kinds of metrics and, while today does only include CPU and memory, we do want to extend these, but this is what we have today and what is stable today to be used, and there is a canonical implementation for this, because the resource, metrics API is actually important for kubernetes cluster to run by itself. So if you do things like coop CTL top or you want to also scale based on CPU or metric on memory, then you can do this consistently in every kubernetes cluster.

A

And you don't need a monitoring vendor to do this and before we actually implemented this, we did the calculations and said: ok, kubernetes guarantees a scalability of 5000 nodes with an average of 30 pods per node, and if we have about 10 metrics for each of those pods, then we end up with and divide that by the collection interval, which is every 60 seconds. The metric server would go and scrape all of this information from the nodes.

A

Then we end up with 25,000 metrics ingested per second, which keeping this in memory is a very, very low amount of metrics and yeah. So we went ahead and implemented this and it performs really well- and this is now the default implementation of the resource, metrics API. In theory. These vendors could still go ahead and implement these. This API, but important is that when you go on any kubernetes cluster, we can be sure that everything works, because this is the default thing that is deployed.

A

For example, gke employs this mini coop deploys and pretty much all of the other vendors of managed kubernetes clusters. Do this and of course, also most of the installation tools do this if they don't shoot them a pull request, because this is actually a requirement of bananas cluster and then the custom metrics API was actually the thing that we really wanted out of this pipeline.

A

We, the semantics, is, are the same as with the resource metrics API, which is you get the latest sample of that time- series that you're requesting, but because this is custom by definition, we don't have any defined metrics for this, and because it's very specific to how you produce your metrics, there cannot be a default implementation for this, so there must be an implement implementation by your vendor for your monitoring system and with the custom metrics API.

A

Each of your metrics actually must be related somehow to a kubernetes object and we'll see later what that actually means and because Prometheus was is so popular within kubernetes and I. Guess that's why everyone is here today we started with Prometheus being the example implementation, but there are other implementations for other monitoring systems out there, such as stackdriver and others, and this is something very, very new.

A

This is only an alpha where, whereas the other two api's are in beta and are available on all kubernetes clusters, the external metrics API is very similar to the custom metrics API. However, it must not be related to a criminales object. What this is useful for is when you, for example, have some service that you're consuming by your cloud provider. Let's say they have some sort of a message: broker that you're consuming and there's a queue length metric that that system exposes, but it doesn't run in your kubernetes cluster.

A

You can't so you can say my pod has this metric. So that's what this use case is for. This is an alpha, but there are already some implementations out there for this. So now that we understand what auto scaling is on an abstract sense and we understand the current metrics api's around kubernetes, let's look at the concrete example with Prometheus, so to make sure that we all understand how Prometheus works and then how this entire system works as a whole.

A

Let's have a look at how Prometheus works, so in Prometheus you, your application, keeps its metrics in memory, and this has very little overhead and basically, in this example, we have started our application and it just just started so there were requests count. This has not.

A

It hasn't received any traffic, so requests count is zero, so Prometheus is configured to scrape those metrics every interval in this case- and this is the default every 15 seconds and when it does that it takes the metrics, your application exposes and writes it into its time series database and then we actually add our application to a load balancer and it receives some traffic and then the internal request counter is counted up and then the scrape interval has passed and Prometheus will again scrape this information, insert it into its time series database and basically do that for eternity.

A

So, let's make this use the HPI. Let's make the HPI useless. So in order to be able to accommodate the features that we wanted to, we had to create a new version of the horizontal part or a scalar object, because if you remember, we were only able to specify the CPU utilization.

A

So the first couple of fields that you're seeing here are actually identical to the first version.

A

However, the very thing that we wanted to change is now an array of metrics, not a single metric, and we can specify multiple metrics about multiple types in this case I'm showing something where I'm saying every pod in this associated with this object exposes a metric called HTTP requests, and actually the adapter gives us some help here, because the actual metric that my application exposes here is HTTP requests total, but I'm not interested really in terms of scaling about the total requests, I'm interested in the requests per second.

A

So it knows which ones of these are counters, basically converts those, and then we can say what we want our target value for each part to be so when this target value has exceeded that we spin up more replicas of this in order to cover our demand- and there are two types of thresholds we can set here. One is a is an actual value, so in this case it makes sense to use a value, but for something like CPU or memory utilization. We would again probably want something percentage based.

A

So when the when the HPA actually does does this request, the way this works is that these api's are what we call aggregated api in kubernetes, and what this means is that when we create these actual instances, so obviously there needs to be some application serving these api's. It registers them with the kubernetes api server and when then the HPA just asks the kubernetes api server. Give me that value from the resource metrics api, the api server actually just basically proxies that to the actual instance of the metric server.

A

If we're requesting the core the resource, metrics API and as we said earlier, the metric server then goes and or actually periodically goes and scrapes these metrics, and in this case it will just immediately return whatever at pass in memory about that requested metric now with the custom metrics API, this is a bit more complicated. You need your actual monitoring system. If you use something like perme, you may be running that within your actual cluster as well or not with.

A

If you use some vendor, you probably are not running that within your cluster, but it's outside of your classroom.

A

Then you, your custom, metrics API, would call out to that vendor and then return that information and prometheus has all of this information because it goes through your I mean you need to set up Prometheus correctly to do this, but it's pretty easy to do that and you scrape all of these pods and the metrics that they expose and that way, then, when you, when you'd, perform a request against the custom metrics API, it will perform a query.

A

Basically, it is a translation layer which converts the custom, metrics request into a Prometheus query and then queries Prometheus and returns the result now, in terms of the result, the example that I just showed how that actually internally works is that when we say we want the HTTP requests rate of all of my parts of this associated object, then I, then what the custom metrics adapter does here is.

A

It does a rate so the inner function we can see here on the metric called HTTP requests total and it makes sure to add a selector here to only select those parts that are actually relevant right now, and it applies range courious. What we call this in Prometheus, which means I, want to see the rate over the past five minutes, and then it sums all of those by pods. That's more, not wouldn't necessarily be important for this query, but in general it doesn't hurt, because if we have, we can do some additional grouping.

A

When we specify the metric, we want to query and basically we could do something like requests per route or per method, and then we want to scale based on that so yeah. So, let's, let's see this in action. I have.

A

Obviously my blog running on kubernetes and.

A

Let's see nobody here in this room has done any requests against my blog Sam.

A

So let's put some some load onto this, so I have prepared something.

A

That will put some load onto this and I.

A

My HPA object looks very similar, actually exactly like what I just showed in my example, so I'm now putting some load onto my my blog and it creates metrics for all of the pods that are running my blog and hopefully, as we can see. Actually the object that I applied has a max of 30. In case you all start ddossing, my blog.

A

So hopefully we should now soon start seeing that the traffic increases for my blog and then eventually that epic replicas are increased. Because of that. So we can already see that the traffic increases and the horizontal part autoscaler actually has a cool-down, so it only scaled periodically. So it doesn't when, when there's one spike up or down that it immediately tears down all the pods or immediately spins up 50,000.

A

So it does this periodically and for now we'll go back to the presentation and then we'll check in later to see that this actually actually worked. Hopefully, if not I have a recording. So now that we've seen all of this of the system, what do I see? What do I already know is going to happen in this space and the in the near future, after that, I will make some predictions of what I believe is going to happen.

A

So the reason why we've been doing all of this work is because we felt heaps err was not was not efficient enough of a solution for this, so we just last week have started formally the process, the process to formally deprecated hipster and pretty much within SiC instrumentation. We believe that this is the correct thing to do, and actually, as of kubernetes 1.10, all of the critical things for a kubernetes cluster to operate have been migrated to use the resource metrics api.

A

So we felt now is the time now that we have migrated everything now's the time to deprecated the system and one and as I already mentioned, horizontal part. Auto scaling is really the default thing to do today in kubernetes, but it does it's not always the right thing so for databases, for example, it's very often important to still increase vertically.

A

So there is a lot of work going on with this, but there's no, the all of that work is still in an alpha and it only supports Prometheus because it turns out a vertical part. Auto scaling is a lot more complicated and a lot more fragile. So there's a lot more thought that needs to go into that as opposed to. We have some rate limit per per pod, and we just want to increase that all the time, because this would involve spinning up new pods and things like that.

A

So there needs to be some more thought put into it, and something really cool personally I think is that very soon in criminales, 111 we'll be able to scale custom resource definitions in kubernetes. So in case you don't know, custom resource definitions are a mechanism in kubernetes to extend the kerbin ids api. So, for example, I am the maintainer of the Prometheus operator, which allows you to create Prometheus objects within your kubernetes cluster, similar to how you create a deployment in your kubernetes cluster, and this will allow us to auto scale.

A

Those kinds of resources and I think that's really exciting, because it means that we can very natively, create these extensions to the kubernetes api and have them behaved exactly the same way as all the other resources and I just want to give a shout out to Stefan and Nikita, who have been really hard of hard at work. Doing this. So thanks. Thanks to them, we will be soon able to do this, and one of the one other thing that we're working on within SiC instrumentation is over the past year or two or three.

A

We have realized within seconds fermentation that the metrics that we provide in urban Eddie's are not as stable as we would like them to be. So there are a number of proposals out there in order to improve this state so that we can have a better confidence in the metrics that we're using today we'll be there tomorrow.

A

Yeah so I think, while that's not super exciting in itself, I think it's really important for this entire ecosystem to be able to share all this knowledge, because if a metric can go away from one day to another, then it's basically useless.

A

So those are the things that I know are going to happen now. Some predictions that I think would be cool to happen. But I don't know if they're going to happen so I I think it would be really cool because we can auto scale custom resources.

A

It would be really cool if we do the use the work that some of the companies have been working on, where they have where they describe a cluster with custom resource definitions such as note pools that we can use all of these primitives that we have and the mechanisms that are available in custom resources that we can automatically scale our clusters with those primitives and today the cluster autoscaler is a separate component that actually does about a lot of the very similar work.

A

As the horizontal part, autoscaler does so I'm hoping that all of these projects will one one day be combined, and this goes very very much into the thought of stable metrics, but is a more broader thing happening in the monitoring space, which is a standardization of all this monitoring. So there's a an effort called open metrics, which is standardizing the format. Basically, that Prometheus has. Obviously there are some some changes, probably going to happen to it.

A

But it's based on that and what's really exciting, is that all of these monitoring vendors are coming together and working with the Prometheus team and other open source in order to create a standard that all of these monitoring systems can then use because all kubernetes components are already serving Prometheus style, metrics, so I think that's really cool and in within our bubble, more or less. There's. Also, some really awesome things happening like G RPC, while HTTP metrics are already pretty standardized in the Prometheus ecosystem.

A

G RPC gives us even more structure as to how our HTTP handlers are registered, and that gives us the possibility to create dashboards, alerting walls and auto scaling mechanisms that work across all of our organizations, so I think that's really powerful and together with that service. Mesh really puts brings this to another level where every single pod within our infrastructure exposes these metrics and they're all they're, all the same width across our entire cluster. So that's really powerful for observability.

A

So really what all this boils down to is we can build reusable alerts, dashboards and why we're all here is to build reusable auto-scaling and share all of this knowledge that we have running these applications that are now exposing very similar metrics in order to serve our customers and fulfill our service level objectives. So with that, let's have a look at our horizontal pot.

A

Autoscaler again, let's see if someone totally broke everything, nobody did cool, so we now have more replicas of our of my blog isn't that cool and we can see that we have sort of a stable request line, which is around 2000 requests per second, which is exactly the thing that we've specified yeah. So that worked awesome and with that. Thank you.