Cloud Native Computing Foundation Cloud Native + Open Source Virtual Summit China 2020, 11 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Cortex: 1.0 and Beyond! - Goutham Veeramachaneni, Grafana Labs

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Deep Dive: Cortex: 1.0 and Beyond! - Goutham Veeramachaneni, Grafana Labs

Cortex is a CNCF project that provides horizontally scalable, highly available, multi-tenant, long term storage for Prometheus metrics, and a horizontally scalable, Prometheus-compatible query API. In this talk, we will talk about everything that led to the recent 1.0 release of Cortex, what it means to the project, users and community. We will then build upon that, and talk about all the new and exciting features the Cortex team is working on, including the next generation storage engine based on top of Prometheus TSDB and Thanos. We will cover why we're building the new engine the scaling challenges we faced and how we fixed them.

https://sched.co/cpB2

A

Hello: everyone thanks for tuning in to the cortex deep dive. I'm super excited to talk to you about everything that happened since the 1.0 release and beyond. So before we dive into it a quick introduction about myself, I'm gautam, I'm a software engineer at tefano labs and I'm a prometheus and cortex maintainer.

A

I've actually been contributing to prometheus for nearly four years now and about two and a half years ago I joined grafana labs to work on cortex, so we use cortex to do a hosted prometheus service at the funnel labs and between prometheus and cortex. I've also worked a little bit on loki, a lot open source log aggregation system built on top of cortex inspired by prometheus anyways.

A

So, let's see what the talk will contain. So initially I'm going to give you a very quick introduction of what cortex is. What use cases does it solve, and then I'm going to talk about the 1.0 release? What does it mean for our end users and community? And then this is uh the next topic.

A

The block search engine is something I'm super excited about and I would like I can't wait to talk about it and finally, I'll share some of the other things that are on the roadmap, and then we can conclude the talk all right, diving in so an introduction.

A

So this is the cortex deep time, uh so I am assuming most of you are familiar with prometheus, but for those of you who are not so this is just a prometheus standard from ethious setup. uh Basically, prometheus is collecting metrics from nginx golang like an application and a database. It can collect a metric from anywhere, and this is like this looks like a typical application in a company.

A

Now you can, you can alert on it. You can graph on it uh on all the metrics that you require and this works really well. For example, if you look at the dashboards, uh this is how the typical dashboard would look like, like you, have the qps and latency for different uh layers.

A

Now this works really really well uh until you have to scale and typically uh there are problems if you scale beyond one prometheus, but why would you need to scale beyond a prometheus? Why can't you use a single prometheus for everything?

A

One, a prometheus is not a horizontally scalable system, but it scales really really well on a single node. Having said that, there are other reasons you might want to use multiple prometheus servers. For example, if you run your application in different different data, centers or different network partitions, you need to run a prometheus per data center. This is uh this is just because prometheus needs to reach all these applications to collect the metrics, and it's also a best practice to run prometheus close to the applications that it's monitoring.

A

So this would mean if you have three data centers, you have three prometheus servers running and this actually works really well. uh For example, if you get an alert you can kind of uh in the alert you will know which prometheus sent it, and then uh you can point your trafana to that prometheus server.

A

For example, if you look at the dashboard, this is how the dashboard will will evolve. So, basically, there are three different data centers and you can just select the data source and the moment you select the data source, the metrics and graphs, for that prometheus server are visible to you. So if you get an alert from you west one you get, you just go to your grafana say: hey, show me this dashboard for eu west one, and you have your information now.

A

This works really well until you have to combine the data between different prometheus servers. What do I mean by combined data? For example, you will be via this solution. You will be able to see the qps per data center, but what, if you want to understand what the globe global qps across all the prometheus reverses, so that's tricky, but it still can be done with just prometheus.

A

The idea is, you run a central prometheus, the federation prometheus, which kind of scrapes data from the leaf prometheus servers. So this has several challenges, the first one being you need to open up, firewalls and port so that the central prometheus can reach the leaf prometheuses.

A

But the bigger problem is, uh if you collect all the data from all the prometheuses to the central prometheus, this will be putting more and more loading into the central prometheus. For example, if you have three prometheus servers, all scraping a million time series, the central global federation prometheus, would have to spread three million time series and this typically starts breaking down, and it doesn't work that way to make this scale what people do is they only scrape certain metrics or even pre-aggregated metrics?

A

So in this case you would only expose the qps for prometheus to the global federation's prometheus and then because you have three different qps servers, you just do qps. You just do a sum of all of three and you get the global prometheus uh global qps.

A

So this works uh really well. If you know in advance what you want to kind of aggregate together, if you suddenly want to do an ad hoc aggregation across prometheus servers, this doesn't uh really work. So the alternate way to solve this global view problem is to use cortex. This is one of the main use cases of cortex. There are several other use cases um in this case uh you run a central cortex. Cluster cortex is horizontally scalable, so you don't need to do any aggregations.

A

You just send all the samples that prometheus scraped, the central server- and you just point your grafana at it now, because the central cortex server cortex cluster has all the samples you can basically write a query that combines data from all the different prometheuses.

A

We have a standard uh protocol called remote right. You just add a few lines of config to your prometheus servers and all of them will be sending data to a central cortex cluster.

A

So this is just one use case. There are several other use cases, uh but this is a good introduction to cortex, I think so. To summarize, uh or to give you a high level view. Cortex is a horizontally scalable, highly available long-term storage for prometheus.

A

It gives you a global view, like essentially it combines data from different prometheus servers. um It you can store data long term as in people store, terabytes and petabytes of data across three years or four years now, and one other key fact uh like key feature of cortex is that it's multitenant. So essentially, we run cortex with three thousand plus uh tenants, uh but it's a single cluster that does all of that, so that so this is a high level overview.

A

So just a quick history of what's been happening, so it was started in 2016 and then about two years ago we started focusing on query performance and scalability and like in this, in a span of six months, we made our queries. 10X faster. We've done a lot of improvements, a lot of caching and it was actually working really well at a really large scale for like queries that touch months of data across hundreds of thousands of cities but uh and then, like later, it was donated to the cncf sandbox.

A

And after that we started focusing on ease of use and community. We started adding more and more end users. We made sure you can run cortex as a single binary and other things so that it's easy to get started and manage. We launched our website and quite recently about three months ago. We did our cortex one dot, release something that I will talk to you about.

A

So if you want to know more about the cortex architecture, how it works and why it was architected that way, I would definitely recommend this uh cortex 101 talk given uh given uh at kubecon north america uh uh last year by chris. uh I believe the talk in the slides, I'll share I'll be sharing. These slides so feel free to watch these uh watch the talk to understand more about cortex cool, so we've cut version 1.0 a few months ago. So what does that mean?

A

So this is the one auto release uh was there. Well, there are a lot of new features in this release. Actually, not so, by the time we cut one dot or release, we've been running cortex in production for more than three years, and we've added a lot of different flags and config options. Caching options a lot of different things uh to make things scale and work, and we've gotten really really good at managing cortex and cortex has been really really stable.

A

We can still read the data we've written three years ago in cortex, so at this point we decided- okay, uh cortex has been stable, has been running in production at scale for a long enough time. Let's get a 1.0, so as part of 1.0, we went back and looked at all our config options and we were like okay. We can simplify this.

A

We can rename this for consistency as uh and we've kind of changed things added same default, so it's easier to manage and you get a very solid conflict just out of the box, because the defaults have been changed, so this meant a lot of breaking changes in the 1. release, but because we made all of these changes with all the experience we had running cortex and production, we also provide stability guarantees as part of 1.0.

A

So there's a lot of stability guarantees, but the main thing is flags and config options. uh So essentially, if you start cortex and if you uh kind of tune context to your needs- and you should not be afraid to upgrade to the next version- because everything should just work uh so that's like well, that's the flag, compatibility and config compatibility is one of the guarantees that we provide.

A

The other thing is reading old data, so we we will ensure that you can read data uh well, that is, or that has been written nearly two years or two years old. We you can. We can still read data, that's more than three years or three years old, but the guarantee we are going to provide is. But if you run a cortex version today, uh it should be able to read the data that was written two years ago.

A

uh The other thing that we strictly adhere to is the api compatibility. So if you build uis or tools around cortex, just by upgrading, we will not break any of the apis that we mentioned here, so you can upgrade with ease.

A

So this basically generally means that a lot of the large users can start using uh cortex in production, uh with all the stability guarantees that they expect of a mature product or a mature service cool. So that's about 1.0!

A

You can read more about all the different things we've added in 1.0 and the different things we've changed, but more or less it's about stability and making sure that we designate it as production, ready cool with that. I kind of want to talk to you about the block storage engine. So this is something that is that I'm super excited about, uh because this kind of represents the future of cortex.

A

So cortex today uh requires two different storage engines. One is a chunk store and the other is the index store. So the idea is uh in cortex when you receive samples for different time series. We basically chunk these samples up compress them together into like a large blobs and then store these blobs.

A

uh We store these blobs in the chunk store and basically we have an index to look up these blobs and then decode and run uh prom ql. On top of this, just to give an example, if you have a time series up instance equals a, we basically create 6 to 12 hour chunks uh of data, and then we store these chunks in gcs or like an object, store and then for each chunk stored in the object store. We create an index entry in the index store, for example.

A

uh Basically, this is how the index looks for each key value or for each label. We basically have a list of chunk ids uh that ex that exist, and then because we have the chunk id, we can go to the chunk store and look up the actual raw bytes and raw data based on the chunk id, for example.

A

If you're, given a query that says, oh give me all the data, for instance, equals a you basically look at the index and say okay instance equals a belongs to chunk number one chunk number three and check number five. Let me look those up.

A

Similarly, if you're, given a query that says, oh give me all the metrics for up in for the metric up with instance, equals a. We basically look at the index uh retrieve all these uh chunk ids and, for instance, equal say we retrieve all these junk ids take the intersection of these two to get the actual chunk ids that match that query.

A

So this is this is over simplification of the index, but this is basically fundamentally how it works and we need a nosql store for this, because the number of entries in the index are super large and the size of each entry is super. Super low and sometimes the index can grow to terabytes, uh which is why we were looking for a nosql store, but there are a lot of challenges uh with the new x inverse sql store, so we support bigtable, dynamodb and cassandra.

A

Bigtable and dynamodb are expensive uh if you're running on gcp, you typically run with bigtable, and we've noticed that in some of the clusters, more than 50 percent of the cost of running cortex was just big table and if you actually wanted to move to cassandra and manage for the nosql store ourselves, cassandra is another distributed system to manage and it's actually not easy and there's a lot of failure modes.

A

So there's a lot of challenges with the nosql store and as we were looking at how we can improve the cortex experience, you were thinking about what, if we remove the nosql store and store the index as well in the object store. So as we were thinking about this, suddenly something amazing happened.

A

Somebody from the community just submitted a large pr uh that basically added uh a support for the tsdb uh prometheus tsdb engine into cortex. So basically, this is thor who was working at distillation, who was like? Oh, I think we can actually add just an object store in engine into cortex and he submitted a pr.

A

We worked with him to make sure it's merged and we're super excited, because this this is what we wanted to do, and one of the great things is that it's built on top of the piano storage engine, so basically uh thanos also leverages the object store for storing the index and data, and we are basically reusing the storage engine in context.

A

There is an amazing talk by prom con. Just google sharing is caring. The talk will be linked in the slides uh as well uh of how basically, uh while working on the block storage engine, the thanos and cortex community collaborated together to make sure that there are improvements uh during it uh like contributed to cortex by the thanos team and to thanos by the cortex team. This was given uh again the prom con and I'll link the talk in the slides.

A

So this is how the cortex block storage engine looks like. uh So, essentially, you get all the samples uh to the distributors that are then written to the injustice, and once the investor has a large enough block, it writes the data into an object store and then we also have a compactor that takes different blocks and compacts them together to make even more efficient blocks.

A

So that's the right path and you can horizontally scale the right path like this there's no limitation there, uh where you can't scale the right path out, and similarly, we have a horizontally scalable, read path uh completely more or less stateless, uh where we basically have the store gateways. uh That kind of download a subset of the index and the query then uh queries the store gateway to kind of see which parts of the chunk it chunks it needs to download from the object store.

A

uh This is similar so basically, instead of storing uh the index in a nosql store, we are basically storing the entire index uh in the object, store and the subset of the index in the store gateway. Here. This way we don't need a nosql store and the store gateway is horizontally. Scalable. The object storage is typically horizontally. Scalable, the courier and query front end are more or less stateless, so you can also scale them up and down as you wish.

A

So we have a horizontally scalable and decoupled, read and write paths and the only uh external dependency is the object store, and this is working really really well, and one other thing that I want to mention again is that we've built this on top of the thanos storage engine, with collaboration from the thanos community, for example, we are using the nano shipper, the thanos compactor and the thanos bucket store gateways in cortex to make all of this happen.

A

So this is working really really well uh we're seeing massive improvements uh in performance, something that I'll share with you today and also a huge decrease in cost, like sometimes even three times less cost, to run the same to store and query the same amount of data, so the status today is basically, we have some clusters uh that are running in the cortex trunks engine that are in broad and we've actually started teeing right. So we are.

A

We started writing to both the chunks and block storage engine, uh the two different clusters, uh so that we can compare query performance and we build this query t component to uh basically query both and compare performance so, but so essentially, whenever somebody at grifana queries our internal cortex cluster, we're actually querying both the chunks engine and the blocks engine which have the same data and comparing the results as well as performance, so just to share something. So the green color lines are the block.

A

Storage engine and the yellow color lines are the chunk storage engine. So this graph is not super clear, but the block storage engine is typically faster than the chunk storage engine.

A

To make this even more clear, so we are basically looking at the 99th percentile of the last week of queries here uh in the blocks cluster and in the chunks cluster and in the blocks cluster. It was taking 450 milliseconds, while in the chunks cluster the 99th percentile was 2.5 seconds, so the new storage engine is already five times faster and we are super excited uh about the whole performance thing. uh But we also have about two years of data in the chunk storage engine which we need to migrate to the blockchain engine.

A

If, when we start to use it so we've started actually onboarding some of our newer customers onto the block search engine in production and it's it's working really really. Well. We've also put in a migration path so that anybody in the junk storage engine can migrate to the block storage engine. If something's wrong, they can roll back to the chunk storage engine.

A

uh So we've put in that- uh and we are starting to test that migration path and starting to move our customers over slowly and that is working also really really well, and once we've actually tested the block storage engine at scale uh for a large number of tenants and for a large number of active time series, we will be calling it generally available and production ready.

A

We are hoping that will happen in the next three months fingers crossed so that that's just a small snapshot of what's coming a completely new storage engine, that's much faster and much cheaper to run, and this engine is also enabling other things uh in the cortex uh project and I'm gonna give you a brief snapshot of all the other features that we are working on, not just the storage engine, so the roadmap.

A

So this so we have a roadmap page now. So if you click on this, it will take you to the complete roadmap that we put up on our website. If you're interested in the entire details, I'm only going to highlight a few things in this talk.

A

So first thing is per tenant retention, so in the junk store, all the data for all the tenants was in a single table and if you wanted to kind of enforce pertinent retention and cortex saying okay tenant a gets three months of retention and tenant b gets 13 months of retention. It was not really possible. You still had to kind of delete partially the table and that cost a lot of right throughput and a lot of cpu spikes.

A

So it was not really possible in the chunk storage engine, but in the block storage engine we actually put each tenant under a prefix inside inside the object store. So in this case, if you look at this, this is the tenant id and you can already apply rules at the object store level to kind of delete objects uh based on retention.

A

But we also want to uh not leave this to an object, store level configuration but would pull it into cortex itself so that the compactor automatically deletes blocks uh when the retention is set- and this is super easy to do with the blocks engine. So pertinent retention is coming. uh We have gotten a lot of feedback that, while cortex is great getting started with, it is kind of hard because there are not enough film charts, another packaging.

A

So this is one of our biggest priorities. We want to clean up our official helm, chart and publish it, add support for all the providers, gcp, aws and on-prem, and we also want to focus on rpm and debian packaging. uh Basically, for non-kubernetes environments, you don't need kubernetes to run cortex. uh If you have communities it becomes easier, but you don't need that um and we want to encourage more and more people who don't run, who don't yet run uh kubernetes to run context as well for their prometheus monitoring needs.

A

We want to add exemplar and metadata support. I'm only going to cover metadata here because of the lack of time so metadata as in so essentially, if you have a metric, you kind of want to understand. Is it a gauge? Is it a counter?

A

Is it a histogram, and you also want to understand what it means in most cases, it's easy to see easy to guess what that means, but sometimes or if the metric is not named properly, it's extremely hard, but the prometheus exposition format actually exposes this and nowadays prometheus stores this metadata and lets. You query it. For example, if you are in the explore mode with the prometheus data source- and you are looking up a metric, you will be able to see the type and help information as uh in the autocomplete.

A

Now this is being done inside prometheus, but we also want to take this metadata and write it out to cortex and other remote systems and add the metadata feature into cortex directly.

A

Similarly, exemplars is one one other feature that we are working on: I've linked the what what examples are and what the metadata support is in the slides um there. This is just a small snapshot of the other features that we are working on and there's a lot more features to come. The first thing is the authentication and uh standard authentication building system so that multitenant organizations can get started with cortex. Much easier soft multi-tenancy, so today, if you, uh if you belong to a tenant day, you can look at all your data and you can.

A

You cannot look at any other tenants data. So it's like you get an all or nothing, but so the problem is a lot of these companies. A lot of companies want a super user view, basically, where there's an admin user that can look at data across different tenants, uh but if, if, if there's a team, the team can only look at its own data, so this is one of the big asks that we have that we are going to uh that. We are working on or like that.

A

We are going to work on a historical import of data. This also is enabled by the block storage engine, and we have a lot more scalability improvements in the pipeline, so just to quickly recap. um So we've talked about cortex, uh the 1.0 release, the block storage engine and the roadmap, so the blockchain engine is completely new storage engine. That's much faster uh on much faster and cheaper to run, and then we also have not just that, but a ton of other things on the roadmap.

A

So if you want to scale out your prometheus needs or if you want a multi-tenant prometheus system, just look into.

A

B

Okay: okay, thank you, gotham. Thank you for your introduction of the context. Cortex. Okay. I it's time for online q, a.

A

Hi folks, uh I already have a question um thanks jerry, so the first question is: will it be easier for consumption to have cortex or thanos features to be included in prometheus or the other way around? So that's.

B

A

Nice question so the idea for prometheus is that it it's just prometheus. You should be able to get your alerts and monitoring and metrics from just prometheus and should not depend on network that way.

A

Prometheus can be much more stable, but if you want to go beyond 15 days or beyond just alerts, uh you can look at cortex or thanos to actually help you solve other problems, but prometheus in itself should be a super stable system that doesn't depend on network so that, even when everything is burning, if your network is down, you can still access your metrics and look uh get alerted.

A

So I hope that answers the question. um Okay,.

B

I see another one in the chat box.

A

Yes, uh so this question is any horizontal scalable prometheus implementation with cortex, I like so cortex- is a horizontally scalable prometheus implementation. But to give you a couple of use cases, um we have grafana labs and we work who provide a hosted prometheus service where people just sign up and send send us their prometheus metrics, and we handle hundreds of millions of active time series uh and like petabytes, of data of all our customers and all of them is that is done by cortex, but some other use cases are.

A

There are companies like gojek and revei?

A

I will link you some of the case studies in the q a section, uh but these companies- uh these are large companies with several teams using prometheus and they want to provide a centralized, hosted prometheus service for their internal teams and they're using cortex. To do that and because cortex is multi-tenant, you just run a single cortex cluster for all your teams, but each team gets their own tenants so that the data is secure and one team cannot look at another team's data and you can enforce multi-tenancy and limits.

A

So, even if one team is sending you a lot of queries, you you can put in limit so that this one user doesn't degrade the system for the other users. So that is some of the use cases. I will also link the case studies section in the in the chat in the q, a.

B

Okay. Okay, thank you. Thank you so much. uh I think it's normal question for you. Okay, thank you gotham! Thank you! So.

A

B