Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Thanos: Cheap, Simple and Scalable Prometheus - Giedrius Statkevičius & Matthias Loibl

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Thanos: Cheap, Simple and Scalable Prometheus - Giedrius Statkevičius, Adform & Matthias Loibl, Red Hat

Thanos is an open-source CNCF Sandbox project that builds upon Prometheus components to create a global-scale highly available monitoring system. It seamlessly extends Prometheus in a few simple steps and it is already used in production by dozens of companies that aim for high multi-cloud scale for metrics while keeping low maintenance cost. During this talk, core maintainers of Thanos will explain basic concepts behind the project, its use cases, and tradeoffs. You will learn where to start and how to quickly deploy Thanos on Kubernetes without impacting your existing Prometheus setup. This talk is recommended for those who want to know more about running highly available Prometheus setup at scale with potentially unlimited metric retention with the lowest possible effort and cost.

https://sched.co/Zevx

A

Hey everybody uh welcome to the uh thanos introduction um for kubecon eu in august 2020..

A

um Today we we will introduce you to thomas, but first we we want to make sure that all of you out there are doing doing great staying safe and I hope- or we hope, that that you're still having fun attending this cubecon all right with that out of the way, let's go, let's get into it and, let's start introducing thanos. So I'm I'm matthias lover. I'm a software engineer at red hat uh working on the observability team.

A

There um on various uh open shift integrations, for example, with monitoring um and I'm a contributor to thanos the prometheus operator prometheus and I'm also the meetup organizer for the berlin prometheus meetup here in berlin.

B

Hey, I'm gadriz, I'm a senior d systems engineer at art form I'm part of the performance of observability services, so we provide proper fees and, of course, with us kind of as a service to other development teams. I also contribute to fanus also maintained a small goal library which lets you interact with the grafana with grafana through its apis. So if you want to do that kind of stuff feel free to use it and contribute to it, and I also maintain a blog at get gadgets.blog.

A

All right perfect so, as we said, we're here to talk about thanos right, so first some some numbers about thanos that um that we want to like discuss with you. So we had a talk like this introduction talk last year in uh san diego at the north america, at lubecon and since then, we've grown significantly like in the beginning.

A

The project was started in november, fully open source and um it like is currently part of the cncf sandbox um and we're like actually talking about like incubating or actually a vote on, the toc for cncf spending. So hopefully, by the time you see this, um we we might have already uh reached the incubation incubation stage with uh thanos. But since, since the last cubecan, as I said, we we've still grown significantly right. So we now have like 6, 000 plus github stars, which is an increase of 1 400.

A

um We have almost 80 new contributors, which is kind of crazy to think about. um We also grew quite a lot. We have like basically doubled the amount of users and slack that join our communication and discussions there, and we also have two new maintainers and triagers from now. Seven different companies.

A

On top of all of that, we also made sure that we have a proper governance in place um because, as it attends with some projects, there are some companies really heavily involved with these projects and being employed by red hat. I can actually say that red hat is kind of like really strong in thanos, but we also want to make sure that we don't that we aren't able to overrule other opinions and voices in the community. So thomas actually put a transparent governance in place where per company you get at most two votes right.

A

All of that combined makes for a really great production users of thanos these days. So um we know of 35 companies that have put their logos on the um from the thanos website, but I guess there are quite some more but like these are the ones that we know about right, we're kind of like here to start with prometheus. So how does like prometheus and thales actually integrate um and like we already have prometheus?

A

So why do we like need faunas to begin with right um and they actually integrate and thanos builds on top of of prometheus? So, first of all, why do we even need monitoring by doing prometheus by doing donalds? So we want to collect and process, then aggregate and display almost like real-time quantitative data about a system such as query, counts types errors, um and we want to like process those times and and server lifetimes. So it's it's quite significant to get this right and to to make it work.

A

So with this, I'm I'm heading the mic over to get raised to introducing more to prometheus.

B

So when building funnels, we tried to stand on the shoulders of giants and the giantess prometheus for because has had lots and lots of improvements over the years. So, first of all to get metrics from a road service. We use the prometheus client libraries and and define what kind of metrics we want exposed.

B

So there are four types of metrics and prometheus with all of the data exposed. We need a prometheus to grab all of it, and this is the architecture of prometheus. We will look at prometheus from these four point of views, so we have the compactor which constantly compacts the data to make it more efficient to carry it. We have the curie engine which actually allows you to cure you for the data to send requests to prometheus and get some data out of it. We have the script engine which actually goes to that metrics endpoint.

B

It gets the metrics from it and puts it on the local storage. And finally, we have probably the most important one, in my humble opinion, the rule and alert engine which allows you to uh aggregate some metrics data.

B

So, for example, if you have some kind of curies that do a lot of computation or calculation you most likely want to procure pre-calculate all of that and the rules engine permits you to do just that, so you kind of you are able to create new metrics from other metrics, and the alerting engine gives you the functionality to receive alerts, some kind of messages when something goes wrong. Now we're going to look like how to extract all those components to scale up prometheus here.

B

In this case we have one prometheus node that scrapes four different services on kubernetes that is running on the second node. But what happens if the prometheus that scripts all of those services goes down, unfortunately, but in a lot of cases the prometheus goes down goes down when we need it. The most the typical way to solve this problem is to have more than one prometheus server that scrapes the same services from that kubernetes cluster. But there is a problem. That's different.

B

Prometheus servers will have a bit of different data at different data at different times. Here we have one premium fuse instance and it writes to its own local storage.

B

But, uh for example, what happens if the local storage gets corrupted or we just lose that disk? We will lose all of that data for this problem to solve it. We usually do run multiple prometheus servers that get uh federated into one. In this case, recording rules are used to pull limited amount of data from the clan prometheus instances, and then all those metrics don't jump to the main cluster from which you can retrieve the data. But what happens at a larger scale?

B

It might look like this, and what happens is that the amount of data that you push over the wire is doubled, and this could be even worse if we have uh multiple promucious instances at the bottom on the client side.

B

There's also this problem that, as we keep adding different promiscuous instances on the client side of a central monitoring, cluster will most likely run out of resources, because we need more ram to store, not just ram but, of course, other resources as well to store all of that data, and this is uh where fanas comes in, because we simply cannot scale that setup so impropable in 2017 started this project called fanas, originally written by bartek and fabian.

B

So fanus is a set of components that can be combined to solve the aforementioned problems, so the famous project has some aims defined, uh so the first goal of the finals project is to be from fuse compatible. This allows clients to seamlessly switch between fanas and prometheus.

B

The second goal is to have a global cure review, so we want to have like one point from which we can cover all of the metrics also. Ideally, we would want to have unlimited retention so that we could see, for example, what has happened one year ago two years ago and to see kind of like trends over time, and the final goal is on sampling and compaction.

B

It means that uh fanus performs the same compaction like that's the goal to perform the same compaction that from a fuse does and in addition, uh funness also has the aim to perform down sampling.

B

Now, let's take a look at our federator prometheus setup again um so from the the fundamental problem is that the central prometheus and the monitoring cluster will simply not be able to keep up with so much data as the client side is being scaled up, so, instead of sending all of that data all the time to a central place, what if we had some kind of component which lets us find that data at at query time, instead of like storing it in a central place?

B

By doing so, we can scale up to a bunch of nodes, because we don't have to worry anymore, about a single node ingesting, so so much data. So we have this mystery component and suddenly we can carry from a bunch of profuse instances, but there's a problem. We don't know what kind of metrics do those from a few instances have. So we need another component that sits next to the permafuse. Let's build that component and call it a sidebar sidecar.

B

In addition to responding to those queries, it can also provide us with metadata about what kind of metrics are available at all and by using that data. The central monitoring cluster can make decisions on where to send those queries. We also want to do this to encode and decode the data in the most efficient way possible.

B

We should use something like grpc, so at this point we have an invented funnels. The finance query component sits in our monitoring cluster and achieves the first goal of the finals project.

B

It implements the same from a piece api, so you can switch between properties and harness query very easily and there's also the sidecar which responds to requests over grpc and it proxies those queries into the prefuse instances, and it also has an endpoint which provides us with metadata about what kind of metrics are available at all on those from a few instances, grpc methods are defined in a file called a protobuf, so this is kind of the main service that we provide.

B

We call it the store api and you can see that at the center there are two main methods: the info method, which gives you the metadata information and the serious methods which gives you serious data.

B

It's also worth mentioning that there's a famous deep dive talk at the last coup on san diego. We won't go into detail right now, but you can look at that talk as a consequence of this design. Famous curry is actually fully stateless, so we've solved also another scaling problem. It means that we can scale found as query as we want. We can create a bunch of instances and point the user to any of them. How about scaling the profuse itself just like before?

B

We can obviously create a bunch of from a few servers and have them scrape the same information and have the same sidecar next to them. Proxying all of those requests.

B

We can use the same font as query and put in front of one or more funnel side cards do the same thing as the main cluster, but this time you won't have to send the requests the same request to two or more side cars. So we've made a lot of progress in scaling up our prophecies instance, but we still haven't solved one problem and that is that the local storage of each of the promo furious instance can explode at any time and that will inevitably happen as you scale up your monitoring.

B

So what? If the phantom sidecar could also upload all of the data that promo fuse produces to a central location and in fact that's what it does. It allows us to avoid provisioning, a fixed amount of storage for each of your profius instance, so introducing another component to our fanus fleet.

B

That's the funnel store it's needed, because once you upload all of that data to the remote object, storage, you need to somehow pull it down and to retrieve all of those metrics that were uploaded to the remote target storage.

B

By using the same store api, we can connect it to finance query and that's again one of the benefits of fanus. It's because all of those components implement the same api. So we can mix and match all of those different components together and, as we scale up and down our fauna store instances, because they are also stateless.

B

There's this problem that fana store caches some information about metrics in the remote object storage to make the keurig more efficient in the slides previously shown last year. All of those different caches were not being shared between different thunderstorm instances and that's why we added memcached support.

B

So now we can uh store chunks, information and found a store and like share between two or more instances, all of the chunks information and- and finally, we have that fourth goal: to have down sampling and compaction.

B

We need another component, and that component is called famous. Compact no surprises there, so it takes the the compaction engine out of prometheus and implements the same thing on top of the remote object storage, so you get the same benefits that prometheus provides, but just on the remote object. Storage. Finally, there is this new component that has been recently added.

B

It's called replicate, so there are cases where you might not be able to store some data in some remote locations due to some kind of, for example, law such as the gdpr. So the replicate component lets you copy over uh the sdb blocks from one object: storage to another. To recap: uh there is the finance query which implements the same from a face api and connects the other story.

B

Pi components performs that application and there's the final sidecar, which proxies the request to the promotions and uploads all of the blocks that it produces to remote object, storage, there's fana store, which acts as a remote object, storage, adapter and responds to the same store. Api queries.

B

There's the faunus compact, which performs compaction down sampling and also applies the retention policies and there's also the replicate component that I just told you about so it it acts as a very nice tool and some cases where you want to copy data from one or about object, storage to another, and all of this is very nice. But there are cases where you cannot control the client side, obviously, most of the cases probably all and like all of the cases, you're able to control your own from a few servers.

B

But sometimes you cannot control the oversight and that's where another component comes in. That mafias will tell you about so in.

A

Some cases you might actually not be able to like go from the central prometheus and talk to the individual clusters, but you kind of have like an air gap situation where, like the clusters, are somewhere- and you still want to have this like central place where you can like- have a global view of everything happening right. So for that we we might like think about a new component where we can have these clusters kind of like sent. There send their data to.

A

However, if we just um if we just do that, and we introduce n clusters, so we can like infinitely scale it up or, like like say, have like 100 clusters all pushing their metrics to the central place, we're like ending up in the same situation as before, where we kind of like take down our own system.

A

So in computer science, we actually have a quite nice solution for this, which is the hash ring right. So we can. We can actually take these hash rings and send like from various places the metrics. So we actually take the name of the metric and all its values. We hash that and then that's kind of what we use to like consistently put the data into the individual nodes in the hash ring right, um so we kind of like solve the problem of being able to to handle the load.

A

We can like horizontally scale our system, and we we can at this point just by putting more compute in terms of cpu and memory next to each other. We can kind of like handle more load efficiently.

A

However, if one of those nodes goes down we're still in bad luck, because then we start losing the data of that one node right and to prevent this, we can actually do something which is called replication. So we can all the data that comes in. We can kind of like um in like our case, for example, we just replicate three times. So what we do is we say: okay, this time series, that's uh that should be stored.

A

We take this and we want to store it like on three individual separate nodes, and then we are already able to say: okay. If two of these nodes are accepting our request and storing the data, then we're in a good position, so we might not even care about the third one, which is like the perfect world.

A

We obviously always have this third one, but um we can lose one, and thus we can always be up in just data, even while, for example, rolling out or there's some problem or like we, we shipped a bad deployment and like it, takes down one of the nodes, that's going to be rolled out first, but not to worry. We still are able to um save the data in two of the three replicas right, so that's really nice, um and then we can put that hash ring either into the monitoring cluster.

A

We can even like put it somewhere else and have like a grafana with a query component and talk to that and the hash ring, like all the components, implement the same store api. So a query component knows which phone is receiver. It needs to ask for specific metrics and then conserve the actual requests.

A

Okay, but now that we have introduced this hash ring, which is really nice, we can actually take the same principle of the psych car and, like the side car, the thunder sidecar is actually looking at the file system and every two hours prometheus will write a new immutable block out of the right headlock to disk, and then the sidecar will take that block and ship it to object, storage, and the same is true for the thundercyc for the thanos receivers.

A

The final receivers are essentially a headless prometheus that can kind of ingest data and it uses the prometheus time. Series database, like everything, is kind of the same under the hood, and it will also every two hours cut a block and then take that and upload this to the same object. Storage so like your thunder sidecars might shift to the same and, like all the data is kind of like coming together at that central um place.

A

That kind of like gives you the ability to to store these metrics for a very long time and that's what we call the final receive component. As I said, um there was actually today a talk by lucas from my team on ingesting a million metrics with thanos receive. Unfortunately, that happened before this talk, so you might want to go and check that talk out um as a recording.

A

Maybe you were able to catch it, but um the last thing that I want to like quickly mention is the thanos rule component. So with all of this filled out, we are still unable up until now to kind of like alert on this or like the like new metrics, um with recording rules on top of this entire system right.

A

So that's where the thanos ruler comes in, and that's kind of like the same component that we can know from prometheus, but as a separate component, uh you can deploy this and then this is actually pointed to a thanos courier, so it will actually go through the entire entire same, like steps occurring, the entire system, as any other finance, query component and transparently, be able to actually evaluate rules and kind of like um know when, like an alert, is about to fire and send this to an alert manager.

A

But like the nice thing is that you can do this across all the clusters, for example, so like for every single instance of prometheus, should still alert there. But now, if you see some problem across all the clusters, that's really the beauty. Now that you can take that with the thorns ruler and like across all the clusters, a lot from that all right- let's, let's recap so again, prometheus at the at the very bottom.

A

We already know about the query component of thanos, but now we also added a new component called the thumbnails receive component um to like be able to create these hash rings and kind of like receive a metric sent from a remote place, a remote prometheus.

A

We already know about the star, so we reused that one, because we already had it and it's quite nicely integrated with the receiver and then there's the real component so recapping. We kind of started with this like one application doing it all, and it's really nice and like that's how it should be if you um want to alert um kind of like in a proper fashion, for the like notes and like go so close to your targets as possible, and we had the query engine and these like different other components.

A

um But we kind of like taken the time and split those up. So we now have the query component and thermostat like graphana and like the thanos query component has ui itself can talk to.

A

We now have the sidecar and the receiver that are talking to either prometheus, describes the actual targets or can actually receive metrics from the outside. So that's how we kind of like ingest data um we actually at the bottom, replaced um the local storage with object, storage and then we actually had the compact store, which were kind of like those uh components inside of prometheus, and we kind of made them able to talk to object, storage to compact on those and down sample with the data coming from object, storage.

A

And finally, we had the rule and alert engine and prometheus, and we now have the thanos ruler for that and that can again go with the same api talk to alert manager so really like taking all of these bits and pieces that were there in prometheus and made them horizontally scalable.

A

So if all of this is quite a lot for you, there is a cutter coda which will walk you through the individual steps on setting up found. Slowly get really some some hands-on experience with thanos.

A

If you want to deploy thanos, there are like four projects, uh one of which is cube, thanos, which I'm the maintainer of, and it's super similar to prometheus. um It's compatible with two prometheus and you can like kind of like integrate both of them um and there's a hand chart which we have in the thanos community organization. Now trying to kind of like combine and bring together maintainers of different charts to work on like the single nice chart.

A

That is like supporting most use cases then there's a knight, thanos operator by bonsai cloud, which you can also check out and finally, there's a nice project. The docker compose project for thanos, which allows you to set up basic farmers locally for development.

A

So in the recent month we actually had quite some participants for google summer code and community bridge chiati actually implemented, delayed, deletions and that's already been kind of like done.

A

But on top of that, we currently have five ongoing projects um the summer, so the first one uh being harshita she's working on a per series, metrics retention, which allows you to kind of delete metrics if they get too old, for example, or like in general, delete specific metrics with thanos. Then uh prem is working on enriching and extending the thanos ci with react so similar to prometheus, where we go to a react based ui um same work as being done for thanos and then ucha is working on uh versioning the website docs.

A

So for each uh thanos version we should have like separate docs um that, like are pretty nice, uh pretty nicely browseable and discoverable. Then yes is working on uh per request, query tracking and limiting.

A

So you can kind of like guard um your infrastructure from blowing up with bad queries and finally, sonia is working on improving the katakota tutorials that were mentioned earlier to even add more nice tutorials there for all of you to to work with, and then there were these two um related cube contacts first, one we already mentioned and the second one was on scaling prometheus and how they got some thanos into cortex.

A

So cortex thanos are ever evolving and like stealing, not stealing but like collaborating on um on various components and functionality, um and you really should check out that talk. um It's quite interesting. What's going on there and with that we're done! Thank you and have questions.