Cloud Native Computing Foundation PromCon Online EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: A New Kid in Histogram Town - Björn Rabenstein, Code Fresh & RedHat

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

A New Kid in Histogram Town - Björn Rabenstein, Code Fresh & RedHat

“Bedecke deinen Himmel, Zeus! A new kid is in town.”

That was the beginning of Prometheus's README.md in the olden days of your favorite monitoring system. Prometheus changed how the world was doing monitoring. Now it is time to change how Prometheus is doing histograms.

After a lot of research and even more delays, we finally have a design doc with the momentous title “Sparse high-resolution histograms for Prometheus”. Guided by this document, we are currently working on making better histograms for Prometheus a reality. This talk will tell you about what already works and what the next steps will be."

A

Hello, my name is bjorn. I work for a company called corfana labs which you might have heard about, but for much longer time, I've been working for a project called prometheus that you almost certainly have heard about. If not, what are you doing here? This is prom con right all right. um So, if you know me, you know that I have an infamously favorite topic in prometheus land, which is histograms and once more I'll talk about histograms.

A

A new kid in histogram town, which you might know is a reference to this very first, almost very first commit in the prometheus prometheus repo november 2012 ancient times right. Matt proud himself uh committed this, and the first sentence here is decadent source. A new kid is in town a mysterious reference. I mean in the days of web search. Nothing is mysterious, you just paste it into google, and then you know what it is, but let's keep the suspense here I'll I'll.

A

Tell I'll talk about the first part of this mysterious reference in the mysterious language. uh As the last thing in the talk, so you have something to look forward to now. Let's call it about the new gators in town, that's where I took my title from right. So back then prometheus was a new kid in town, nobody, or at least not very many, had an idea how to do monitoring in that way, um and then prometheus became very famous, and it's not so new anymore.

A

So this is probably why this this sentence is not in the readme anymore anyway. Back then, the interesting part is that prometheus was new, but it was not like unheard of. There were a few people, mostly working at google. I guess that that knew this kind of monitoring and um yeah, and then there was theory about this, how monitoring works and everything. So this is not like prometheus, essentially made this available for for everyone in the open source world, but it it's. It's like.

A

It wasn't fundamentally new that nobody had heard about all those ideas before and that's a similar thing like um fractal self, referential, repetition of history or something with histograms right. So they are super new, so new kid in instagram town, we all have very very new histograms in prometheus, but those histograms are like not really new. They. They are based on ideas that have been around for long for longer than prometheus, actually and um just very basic idea.

A

It's just the challenge is more like how can we make them fit into the prometheus way of doing things? And that's a lot of my recent work about and also um another idea here, is that yeah I've been talking about histograms forever right. So this is not really new, but it's still very new and yeah.

A

So I've been talking about this forever, especially you could uh watch this trilogy of talks. um Interestingly, the first part of the trilogy is newer than the second, it's kind of a prequel, but we know this right. um This is about kind of why histograms haven't materialized, yet the new style histograms uh kind of the history, how we knew that the existing histograms are kind of a prototype, but we got stuck with them for some reason, and then certain events happened that made it even harder to to like follow up with the old ideas.

A

uh So this is kind of this first part, um it's kind of a bit funny recreational. The second part is the most important one as often with trilogies. um It was supposed to also talk about the past. We didn't which didn't happen. So that's why there's a first part now a prequel. So the second talk talks about what we are present state as of 2019 prom gone two years ago uh and the future, what options do we have um where to go?

A

uh So that's very important to understand where we are and then the third part of the trilogy is kubecon last year, two point eu, but it's online and anyway, so you can watch it. um This is essentially after some research, where I proposed what to do quite concretely already, and I also announced a famously I announced a design knock.

A

This that will come out just after the talk in reality it took a bit longer, so this got published in february this year, not so long ago, but here it is it's epic 26 pages or something so totally read it. If you are this type of person, but if you think this is too much effort just watch the talks, um it will be also okay right, so this went through various rounds of review and we discussed it at the prometheus dev summit and agreed that this is the way to go.

A

So this is kind of the the baseline on which we are working. Now I mean mostly it's me, but I hope that I get help. I need help from various experts and also like yeah contributors were asking. What can I do and I told them- I first have to come up with a master plan before we can even do anything right. So this is the master plan here it is okay. What does this talk about? This talk is kind of a showcase.

A

What what already works and what are the kind of yeah very, very imminent, next steps to do so. There there's a slide from the original trilogy. Third talk, but I said I will not repeat those talks and um actually, if you have never seen any of the or like, if you have no clue what I'm talking about right now, it will be also hard to follow the rest of this talk. But this is online right.

A

You can stop this, the stream now and you can watch those other talks or read the designer um or you can just keep watching and then later read it up and understand everything, but in a nutshell, right very shortly. This is how the new histograms look like they have. They embrace sparseness, so empty pockets might happen all the time, but they don't cost anything um in turn. They have an infinite amount of pockets right, so um they they.

A

They look like the same all those pockets, but this is only because the x axis here is logarithmic, so it's exponential pocket from zero to infinity and actually from zero to negative infinity. But I haven't drawn this here: there's a resolution parameter which tells you how many buckets every power of 10 has so after every power of 10. You have an guaranteed like around.

A

Pocket boundary and then the mathematically inclined might already have noticed that around zero you will have an infinite density of pockets. So there's a zero bucket of a certain, very small width or just all the observations in this very small interval are falling into this so-called zero bucket. That's how they look like and that's how we encode them in the exposition format, um so the actually populated pocket are marked down as spans and then within each pocket. We we just have deltas between one bucket and the next.

A

We don't record the absolute value in each pocket. They, those numbers, could grow quite a bit, but the deltas in between are usually smaller because pockets, just like I mean in a very chaotic distribution. This might not work out, but in reality you have kind of smooth distributions.

A

So going from one bucket to the next is usually a small step and smaller numbers. If you do, some kind of fine encoding can be encoded in fewer bytes and that's how it works out on the wire. This is by the way kind of built in into protobuf.

A

That's one of the ideas of protograph, which is also the reason why we're kind of bringing back protobar format for at least experimenting with the new histograms, we'll, certainly not just bring back the old protocol format in the final release code, but like you should notice that openmetrics has a prototype option. So this is probably something more like the direction to go in the end. But for now we just experiment with old prototype format extended for those new histograms all right. um This is enough about old slides.

A

Now I want to show what is already there, um the, as you could already have guessed, the development is kind of in flux, so we don't do this in main. It is also quite invasive, which is another reason why we can't just hide this behind a feature flag or something. So there will be development in a separate branch for now, and the branch will be called sparse, histogram in various repos.

A

At some point, this will be ready to go into main. Perhaps this is the point where we cut prometheus three, but we don't have to decide that right now right.

A

This is what we experiment in a branch: prometheus prometheus is where the meat is right, so it's uh tsdv promql ingestion, but of course we have to start with exposition and we have to pick an exposition library for that and, as I already hinted, we're using protobuf the old protobar format and the only official exposition library that still is backed by the original protobuf data model and format is client goaling. So we just use that one also, we all like go right so kind of nice coincidence.

A

The original prototype format is described in the client model repository, which also will have a sparse histogram branch right. So that's the idea here all right. um Let's look at the changes um incline model, that's really very little. These are all the changes, and now you see how easy it is. You have this pause pocket resolution. You have the width of the zero bucket.

A

You have the count of observation, the zero bucket, and then you have negative and positive sprawl sprockets, which both are just a series of spans with offset and length and the deltas between one bucket and the next, and as said, if you pick the right data type here in protobuf, you'll get va encoding for free on the wire.

A

So this is a very efficient way of encoding those boss histograms without doing a lot of special super fancy, bitwise encoding or something so that's super easy to to get done, and it's already quite efficient and might actually be in the sweet spot between encoding effort and and size on the wire. This is all discussed in detail in the design door, but these are all the changes we need in glide model. Pretty cool changes in client goal link are fairly involved, so I can't show them on a single slide, but I can demo them okay.

A

So here we are. This is a simplistic minimalistic little program that exposes just a single histogram. To do that, I'm not using the default registry. I use a custom registry here so that I don't have any weird default, metrics, just a histogram. You see this is completely standard. This is nothing new yet and the buckets I define. I use the exponential pockets helper here to define 13 buckets and I use I calculate the factor here, and this is the mathematic behind the width of the buckets in the sparse histogram.

A

So I use the old histograms but define pocket boundaries that will coincide with the bucket boundary of the sparse histogram. You don't have to do this, I'm just doing it to demonstrate something I'll talk about in a second register. This now simulate observations, so I simulate an observation: every 10 milliseconds, so it's kind of as if we are serving 100 requests per second from this pseudo service binary or something- and I use random numbers that are normally distributed.

A

200 milliseconds, plus minus 50 millisecond standard deviation so that we get a bit of a distribution right and we run this in a go routine serve metrics. That's it right! Okay, let's look at this. How it will work out, go run, demo, dot, go and already runs right. Let's check this out, let's curl the matrix, endpoint adhd matrix there. It is, I mean so far, nothing new right. This is the classical histogram.

A

Ideally I said this should uh have a round pocket boundary at every power of 10 uh floating point: precision, blah blah blah. It doesn't really work out, but you you get the idea right. You can already see how we accumulate observations right and most of them are kind of ending up in in this pocket. Here uh this is cumulative like it goes up and over and up, but like the delta between bucket is kind of what is actually in the pocket.

A

It's also a delta- um and you see this is such a low resolution that most of the observations just end up in this one bucket here and we are going to change that right. So, let's um go back to my code.

A

Here I prepared this, so I don't typo again mistype anything. So this is the only line you have to add, and I told you I want a default resolution of 32 for these small sprockets right now in the code. The default resolution is the default. Behavior is don't use small sprockets right now. This just adds ball sprockets to the normal histogram, in addition to the normal pockets. So it tracks everything in parallel, so we can look at it in parallel, but this is why I use this interesting bucket layout.

A

If the buckets in the conventional history room are just coinciding with boundaries in the new sparse pocket, the library could totally just use spouse pockets and then render the conventional buckets in a certain way. I mean it can't render an infinite amount of pockets, but it could render like those 13 buckets if you want them based on the data on the sparse histogram, so it only has to track the small histogram right now the code isn't doing this optimization. You just get both in parallel, so you could still do the old stuff. Okay.

A

Now, let's run this one.

A

Yeah what's happening here, stop it run it and, of course, the existing released client calling library doesn't know anything about sports pocket resolution. So what we have to do is use goget with go modules to tell go to fetch something from the sparse histogram branch. um This is the way to do so. Remember this. If you want to play with it on your own, you just do this um and then go modules creates the pseudo version. It's not 1.10.0 anymore, about 1.10.1.

A

It's also pinned to a certain commit hash. You can actually look at go mod to see how this works right. So now it has pinned this here and even it has noticed, but this will happen automatically thanks to what's inclined goaling, that it also pins to a certain version of client model which is not in the main branch, so that nobody's confused was actually using this normally okay. Now it should just work, go and demo.

A

um It seems to work. So, let's curl this again and um as you can see, nothing has changed. This is kind of what's the weight anticlimactic right. So the reason is that the text format is very, very not suited at all to represent those small sprockets in an efficient way. It's all done in the proto format which, as I told you, we are bringing back in prometheus just for this experiment and klein. Golling never left this path of virtue, and so we can just tell to the current day that is possible right.

A

Any go binary that instrumented, prometheus client goaling can um can be asked to serve protobuf, and you do this by adding this little header um hit enter now. Curl is very friendly and tells us this is horrible, binary stuff. If I dump this onto your terminal, it will all beep and go whatever haywire.

A

So it's not doing this and that's cool, um but how do we? How do we look at the new stuff right? So there's a little known secret. um I mean there's the text format of prometheus and there's the proto format of prometheus, but you can actually represent protobuf in a text form.

A

So now it's it's getting deeper right. This is the text format of the protocol format, if you just say, encoding equals text and that's also works with every client golang instrumental binary. You can do this at home if you want right so now you see the protobuf in a very verbose form on the wire. Usually this is the binary form, which is very dense right. So you see here in even more verbose than the normal text format. This is the conventional histogram, and now you get the new stuff.

A

So you get the sparse pocket resolution of 32 the threshold. You see we have two spans here. One is just one pocket like one very low value observation, and then we have 10 consecutive sorry, 34 consecutive markets, 10 empty buckets later, and here all the deltas and you see as promised, they are all quite small. They only get larger around the peak here right and then the minus means this is where the pockets become less populated, again um yeah. This is this is how they look uh in protorov.

A

We can't even see explicit boundaries and everything, because this is all implicitly encoded in the convention. What does the resolution number means, but they they more or less coincide with those pockets? Like I said you could make the live, renderer the conventional histogram at no additional cost essentially, and um the scrapers that want the old instagram could still have this, but you have to pick pockets. You cannot expose an infinite number of pockets. Okay, that's the demo! um Let's go back to uh the slides, prometheus prometheus, that's what we're all interested in right.

A

um So the problem here is that um this is actually the hard part right. I'm currently working on ingesting the protobar format, not in the super optimized form that is, has been implemented for the text format. It could be done by hand coding a prototype browser, but yeah not doing this anytime soon. This is just for experiments, and I said we will. We will find out what's the final choice of how to represent the new histograms on the wire and how to ingest them. uh So that's easy to work on just check out.

A

If you watch this, there might actually be something working in the sparse history and branch. The hard, but also very rewarding part, is to teach the tsv to save those new histograms in an efficient way, and this is where I hope for like these three experts to help and then he'll be dragons. That's how prom ql um should uh do all of this to be like to to make full use of the new histograms.

A

You kind of need new concepts in prom ql, where you actually act on an histogram with a potential infinite number of pockets like yeah. You cannot just represent them all as individual time series from the point of view of promptql but to be again compatible. You can again create this kind of view of an um new, sparse, histogram and and and present it as a conventional histogram, with a selected number of conventional buckets, so that's kind of a migration path, but yeah in the in the the like shiny new world.

A

We have a prom girl that can actually act directly on on sparse histograms and recording rules that can spit out new, sparse histograms and all those things right. So that's really really tough to get right. So um don't don't be too like this won't be done next month or something all right we are done here um and as promised resolution for the decadent himalayas. That's of course, german. I think matt back then was really into german and um what's the greatest literary figure of modern german, that's a person called goethe.

A

Here I like him, I just sometimes would go to easter eggs into my talks. You might not have noticed so now it's very explicit right, but it's not even my estate. It was kind of mad who put this easter egg into it. It's an awesome. Poem awesome mega rant again the gods, mostly zeus, so um you could now think why this what this has to do with prometheus there's one obvious reference. Of course, the title of this poem is prometheus, so perhaps matt just wanted to tell us. I'm all now.

A

Also writing a prometheus like like gutee did everyone should try to promote these ones in their life or something anyway? That was it. um You can see all my talks and how to reach me and we still have q a so see you on the qri bye.