Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Thanos: Easier Than Ever to Scale Prometheus and Make It Highly Ava... G. Statkevičius & P. Saraswat

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Thanos: Easier Than Ever to Scale Prometheus and Make It Highly Available - Giedrius Statkevičius, Vinted & Prem Saraswat; Red Hat

Prometheus is a leading CNCF Graduated project that provides a solution for all things metrics. However, it lacks scalability and high availability features. Thanos turbocharges Prometheus by adding those features while keeping storage and maintenance costs low. In the first part of the session, you will discover how to leverage Thanos in your organization just as many other companies have already done . You will learn enough to continue on your journey toward deploying Thanos in Kubernetes or any other cloud native environment. In the deep dive, you will find out what we have been working on recently to make everything even better - deleting series from blocks, caching queries with the query frontend, making the ruler scalable and stateless, optimizing the compaction of very large blocks, streamlining the receiver, and more!

A

So hello, everyone welcome to our talk in this cube con. It's called thanos. These are the number to scale prometheus and to make it highly available. uh My name is guedras and with me I have brown and.

A

We will talk about thanos, which is a global scalable, highly available prometheus solution, which provides you unlimited, metrics retention. So at first we will do a short introduction about what prometheus is, what scaling issues it has, and then uh we are gonna talk about what has been added to tana since the last presentation at kubecon.

A

So at first let's, we will introduce ourselves.

B

Hello, my name is prem. I am currently a software engineer intern at red hat, I'm working with the observability platform team. uh I have been an oss contributor, like with thanos like from quite some time like uh last year, when I, uh when I started my gsoc project with them yeah. That's all.

A

And my name is gadgets, I'm a central liability engineer at visited that might not be such a familiar name to you. It's just essentially a secondhand fashion marketplace. I am part of the observability team. There I work on metrics traces blogging and I'm also uh open source software can be turned contributor and a fan. So I started in the finance project a few years ago uh as part of my work at my previous workplace.

A

And yeah ever since then I am involved in the project and also I contribute and maintain a goal library for and for interacting with grafana. So it permits you to uh automate various graphand actions in that type safe manner.

A

So if you are ever, if you ever want to do such a thing then go to github.com, grafana sdk and finally, I write about software engineering worthy related topics. On my blog at gearhears.com, and now we will jump to the site about the pandas project.

A

So to give you a short history lesson: it's the project started in 2017 at improvable in november of 2017 by bartek and fabian, and that was open source from the get-go.

A

A

Over the years it has grown quite a lot. Nowadays, it's a cncf, incompetent project, just one step away from being a graduated project and being an incubating project, means that thanos follows certain standards. So, for example, we are vendor neutral.

A

We have a public roadmap, uh we review pull requests quite quickly and so on, and on top of that we have a quite a vibrant community.

A

So since the last cubecon presentation we've gained about 2000 new github stars, we've gained 75 new contributors uh since last crypto presentation, which means that we've grown about two times since 2019, which is pretty insane. If you ask me- and we also- we've also gained 839 new stack users.

A

So if you have a question about anything related to finance, then it's a very high chance that someone will be able to help you not just from the maintainers team and we have lots of maintainers from different companies. So and on top of that we have a transparent governance.

A

So it means that not one of the companies will be able to quote-unquote decor the whole project by pushing it into one direction that they are interested in, and that means that really anyone can come forward with any kind of idea and we will consider them that idea.

A

And, what's really awesome, I think, is that we are part of the prometheus ecosystem. So we don't re-implement lots of code. We tend to reuse code from other projects in the prometheus ecosystem. So, for example, we reused the whole prompt ql engine from prometheus.

A

uh We used the end-to-end testing code from cortex and yeah, it's just awesome and finally, on fanus.web page we have a list of companies which have come which have publicly announced that they are using thanos in production. So you can see that we have really lots of popular names using tunnels. So it's not just the sheer number of users in our community, but also companies just both, but also companies are probably announcing that they're using tablets.

A

That's how awesome it is.

A

So, let's do a short recap on what is prometheus and what downsides does a single note from if you use have and and how far us could solve those problems?

A

Yeah thomas really loves prometheus, um so this diagram shows you the different components which make up a single node prometheus.

A

So we have the rule and alert engine which periodically evaluates expressions, uh then, by using the security engine and the local storage, and if certain thresholds are have been exceeded, then the alert engine sends those solders to alert manager.

A

The query engine is, of course, responsible for the curies graffania and curies come to it.

A

The scrape engines scrapes different endpoints in this case the these are services, one two three and they all expose the metrics endpoint in turn. And finally, you have the compactor, which makes the more efficient.

A

But what happens if that prometheus node just goes down? Well, in this case it's a pretty bad situation to be at and it could be even worse if the local storage of the first node would get destroyed, then you would just lose all of your metrics of that single instance.

A

Of that single prometheus instance.

A

uh How how in the vanilla, prometheus world you would solve this problem? Well, you would scrape the same metrics from one or more prometheus instance, but then it has its own downsides.

A

For example, the same time series would be more or less stored on both nodes, which means duplicated data and, and you would have to retrieve two times more data when curing, which is also pretty bad, and what? If, for example, some of the prometheus instances would not be able to scrape one or more services anymore.

A

Well, then, you would be in a very bad situation in the prometheus world without thanos or cortex. You would solve this issue most likely having another prometheus instance somewhere in what's called a federation mode in which prometheus one prometheus instance is creating matrix from other prometheus distances and then that one from fuse instance, which isn't uh which is scraping the federated nodes, would have to like all of the metrics would have to be transferred over to the central node.

A

And you know when you would just run into the same, because you would have two times the same data and if you would want to avoid sending two times the same data, you would have to use some kind of recording rules and only send a pre-aggregated date over to the prometheus, which is in the federation mode and so on.

A

So with those issues, it's not really it's kind of hard to solve those issues by just using prometheus itself, and that's where founders comes in and prom is gonna. Tell you about how farmers solve those issues.

B

Yeah, so uh we are going to talk about uh first, some features of thanos, so thanos allows you to have global query view over your data, so uh you can run a single query which touches like all of your infrastructure.

B

Like all of your infrastructure metrics, uh we can have unlimited retention with the help of object. Storage uh thanos exposes promoters compatible api, so you just need to you can like uh put those endpoints anywhere where a prometheus query is supported and thanos also has support for down sampling and compacting your metric data so uh yeah. This is what a typical high availability setup looks like. So uh you have multiple prometheus instances running, uh which was which, which are scraping some which are scraping metrics from some cluster.

B

You would run a sidecar along along them, it the components called thanos sidecar and you will run a central thanos query. So, whenever you would query this, uh the thanos query for some data uh thanos query: does a grpc uh api call to uh all the sidecar instances uh to collect those data and uh these thanos grpc these thanos side cars proxies this uh to the prometheus running uh there.

B

Ultimately, all the data get collected into thanos query uh and then uh it the query, gets executed and the data and the result gets returned to, for example, grafana to show those dashboards.

B

Thanos also does like a deduplication on the fly so that if two primitives are scraping the same cluster, the data would get deduplicated automatically now uh to uh to uh add. uh Unlimited retention. uh Thanos sidecar supports uploading prometheus blocks to object, storage, all the sidecar uploads, the blocks to object, storage for longer retention.

B

Moving on the next step, we'll have we, we need some way to uh query this data from object storage. So this is where the third component thanos store, store, gateway comes in uh thanos store uh is a component that sits between uh thanos queria and the object storage. Whenever thanos query does some uh requires some data, uh it would uh send a query uh to the thanos store uh like it sends. A query to the sidecar, and this thanos store would uh get this data from object, storage and return it.

B

uh There is this another component called thanos compactor, which is like a standalone component, but it runs in the background uh regularly compacting and down sampling. All those blocks in object, storage, compaction and down sampling improves query performance across the thanos fleet.

B

Now uh to reduce a lot of requests to uh object storage, we can, we can run a memcached uh server to cache those uh cache. Those uh chunks and blocks uh uh memcacity not only like, improves the performance, but it also reduces the number of requests to object storage which in turn, can reduce uh the cost of uh the whole setup.

B

That's what a typical like thanos high availability setup looks like. If you have any more questions you can uh come on to like thanos channel on cncf, slack or thanos.io.

B

Next, we are going to talk about some new features in thanos, so the first feature we we are going to talk about uh is adding support for deleting particular series is you can always delete a whole block of data by just going on to object, storage and deleting it, uh but uh like deleting a particular series, uh was not possible uh before in thanos, uh so we have introduced a new tool called thanos tools bucket rewrite which, given a block id, can rewrite that particular block.

B

So if you want to remove a time series from from a particular block, you can just rewrite that block the thanos tools. Packet rewrite will create a new block since prometheus blocks are immutable.

B

This block will get uploaded to the object, storage and the older block will get marked as get marked as deleted, which will then get which would then get cleaned up by the compactor.

B

This this tool can uh help you in situations when you want to delete some data uh uh due to gdpr regions or similar laws yeah. uh So the next feature we are going to talk about is query front end uh as gate just uh told you before we are like thanos and cortex uh have a lot of collaboration going on and query.

B

Printen was a component of the cortex project and since it has been like contributed to thanos as well, uh the query front-end is a querying layer on top of thanos queria uh with the help of memcached. It can do. Caching of the queries.

B

Also, whenever query comes to query front end. It splits those query into time, windows into multiple pieces and run those queries in parallel to improve the performance. Those pieces are then like cached into memcacity, uh to uh like to improve further uh query performance as well.

A

And I guess I wanted that not only caches the queries, but it also caches the requests of label names and values.

B

Yeah, so uh those label names and values get cached in there as well, which improves the like overall query time uh from uh data from uh graphing engines like grafana.

A

Yeah so we got a very quick drop down of like label names, suggestions and so on.

B

Those were the major features which we uh had like after the last kubecon talk. uh Now we are going to talk about what's uh what's in store for future in thanos. So uh recently there have been a lot of improvements around multi-tenancy capabilities of thanos.

B

This includes uh making the components tenant aware in a sense so that we can include uh we can introduce features like uh per tenant rate, limiting into thanos or tenant aware resource uses observability.

B

So uh this this also uh in uh this also builds upon the existing multi-tenant capabilities of thanos receiver, uh which is like already multi-tenant uh with this. This comes us to like the next point. uh We we are doing some work towards maturing like thanos receive component. uh This work includes making thanos receive easier to operate. uh So this there has been recent work around splitting thanos received into two components so that it is easier to operate and making it more resilient by introdu, introducing things like shuffle sharding and improving its memory, footprint, etc.

B

We are open to uh all your ideas as well. You can uh go on to thanos, github repo and file a feature request, and uh we also run uh thanos contributor officers every week uh in these meetings you can come and like come to a video call and discuss about uh thanos related projects or, if you are like contributing and you are blocked somewhere, you can uh discuss those things as well. uh We organize these contributor officers uh in both europe and usa time zone.

B

So no matter where you are, you can uh meet us like in in two weeks uh the maintainers uh join these contributor officers regularly to get feedback and answer questions yeah.

B

uh The next thing we are going to talk about is about mentorship programs. Thanos has been continuously participating in uh lfx and gsoc mentorships. uh There has been a lot of uh cool stuff implemented by the uh mentees. uh I was also a mentee for thanos during gsr 2020 and uh uh so like some currently ongoing uh projects uh in in thanos by by the uh as part of the mentorship programs are uh like multi-tenant instrumentation in thanos.

B

So this involves uh uh like answering questions that which tenant used, how much resources uh from thanos- and this is the part of the like- bigger multi-tenant uh movement, going uh going around thanos. The next project is to make the ruler stateless.

B

This would allow making the ruler more lean and like easier on memory footprint.

B

The next project is vertical block sharding, uh so in in bigger deployments. Thanos blocks can increase like a lot in size due to compaction and uh and in those places we somehow need to limit the size of the block and then maybe split bigger blocks into smaller ones, and this is the project on going uh to do to do exactly that.

B

The next project is implementing examples. Api inside thanos, so due to the recent addition uh in prometheus for the exemplars support, thanos is now planning to extend, extend its internal api to support xmls as well. So we can support the same api from thanos.

B

So if you are looking uh into using exam class with prometheus and you are using thanos, you would soon be able to use exemplars along with thanos as well.

B

So uh thank you. That was all uh if you have any.

A

B

Yeah you can ask and uh if we are not able to answer them now, you can always go into like thanos dot io and come on to like a slack channel, to ask.