Cloud Native Computing Foundation Prometheus, 13 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Behind the Iron Curtain: Getting Metrics From the Browser into Prometheus [I] - Tom Wilkie

Description

Behind the Iron Curtain: Getting Metrics From the Browser into Prometheus [I] - Tom Wilkie, Weaveworks

Prometheus has so far been pigeon holed as a monitoring technology for the backend; well, no more! With the rise of single-page apps, gathering metrics from within the browser has never been more important - latency as experienced by the user is much more important than any measurements taken from individual backend services. And if you can’t track Javascript errors that could potentially lead to blank screens, you should feel bad.

This talk presents a method for tracking UI metrics and errors with Prometheus. By getting this data into Prometheus, you can reuse the dashboarding and alerting options that you've grown to love. And by using a common monitoring stack across frontend and backend, you can finally claim the throne of the king of fullstack!

About Tom Wilkie
Tom is a Software Engineer at Weaveworks. Previously he was at Google as a Site Reliability Manager for Google Analytics. Before that he was Founder, VP Eng and CTO at Acunu, and before that a Software Engineer at XenSource. In his spare time, Tom likes to make craft beer and build home automation systems - and distributed tracers!
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Rather than hello again or maybe not again, I was speaking in the other room. My name is Tom Wilkie I'm, a software engineer at weave works, you're, probably getting sick of hearing from weave works. Right now is this one isn't going to be about weave cloud or anything about that this one's going to be about? Actually it is anyway, we have a six-armed, let's get back someone during that time. We have this six client web app running on the cloud.

A

This is probably familiar to a lot of people and this the code in here I'm, not a script developer, so we've got a backup, JavaScript developer in the corner, he's going to look at me funny when I say the wrong words, but this this web app I, went and checked out the 78s 28 thousand lines of code.

A

Do we pay you by the line of code so and- and this originally had no monitoring right if this was a back in that you would have been shouted at to death by now and this talks to we, it is model. This is monitor group from to backend this written ago. We now have this now I'm, really keen on monitoring, gathered that and I'm really keen on using the same tools for all our developers like I. Don't like the fact that there's there's loads, the back end developers may talk to each other I think.

A

Actually, Joe Vader made a really good point in the keynote this morning about how the development experience for front-end developers is. You need code in one window and your your website in another window. Whenever you change your line, it instantly update. Don't have that on the back end, he me right. We need set of tools so about I, don't know two months to go. We set out to try and use Prometheus to monitor the code running in our users web browsers that 28 thousand lines of code running in the web browser.

A

Now this is what used to happen when read wrong with end I'd go on our company slack and I'd our eyes. Can anyone else load the dev environment? No, no I can't let the prod environment either.

A

Don't think this is a local issue and someone would confirm then we'd have an emergency and we'd roll back and everything would be okay, and this is what I really wanted was I want to alert those come up to say it: okay, I, don't I, don't like having to manually, go and refresh the web page to see if it's working. So this is what we wanted. We knew what one of the things that I liked about with prometheus is how good the community is.

A

It's a very knowledgeable community event is in on the topic of monitoring. So a thing I did was fire off an email to Prometheus, saying: hey guys, you know, as anyone else do this already. Is there a current state of the art that we should just copy and can I avoid. You know, can I avoid writing my own code basically and really get any of them.

A

Responding just wrote back to me and said: there's this thing here and there's this thing here, but I don't really know of anyone who does it does what we're trying to do, and so hey I thought right. Great I get to write code now so so yeah. This is what came up with. We came up with a system which is a little JavaScript library which is open. Source here runs in the users web browser, then it pushes metrics and if, if you're familiar at all with with the Prometheus community, pushing metrics will know this gateway.

A

This push gateway. That also does aggregation, which is also a big no-no and then fight and scrape that gateway, and so the rest of the talk is basically going to be looking, but these these two components that we've added that are they plan on doing to make this go better and, and it turns out, there is a state of the art and we've missed it, and so we're going at the end. We're going to look at what other people do so the first bit prom des this, nothing about I, didn't write.

A

This Chapel wrote this this in San Francisco couldn't make it to the conference. I was hoping he would co-present with no person with me, and so yes, I, say the wrong words Dave real correctly anyway. The failure modes for the job think about how a web app works. You actually download the app and every time you go to the web site and if there's 28 thousand lines ago, it's a couple of mega JavaScript, so the first it can fail.

A

Is you can fail to download that JavaScript, okay and the user words enter with a blank page? So that's failure mode number, one also kind of similar failure. Mode number one is the JavaScript can be invalid right, so the kind of hoping that that will be caught by our build chain, because the JavaScript is built apparently trans pilot is that the right word is transpired using using babel and gulp and an or golbez it doesn't use golf anymore. Okay.

A

So there we go so it's we do a whole bunch of work and it lit and it's a unit test some unit tests. Well, there will be more so we're kind of hoping that will avoid the invalid case, but it's still possible that it will sneak through okay in case. Is that there's a bug in the code and you click a button? No there's no bugs in the code, so it's gone really. Red second case is a bug. If you click up renders a new page as far as I understand it uses.

A

This react thing which I've no idea what it is and that rent paid and that render can fail. But that's the second case we want to. We want to capture, and the first case is not maybe a failure mode, but one of the things I'm super keen on, because my main job at weave works is developing. Cortex are hosted. Prometheus thing is like I want cortex to be really really snappy well, I.

A

Think there's an opportunity there for us to produce a really fast monitoring system, so I care a lot about high client server, latency initiative, something as a backup engineer, I can't measure on the backend. This is something that Jordan added last night. So don't say this isn't fresh. So another thing to note about the content code. Is we we compiler? As I said, the acting L is served from our cluster, but the rest of the content. So there's one big javascript file, there's one big CSS file. One image.

A

Don't really have images, apparently that's so 2016 and some fonts and some fonts right so for files get downloaded. Three files get valid.

A

Okay, so will inlined into the assets apparently anyway. So one of the things you'll notice is: oh, yes, so the way we deal with this because we hosted all the assets on s3. This is what I do now about.

A

We host all the assets on s3 and we fingerprint them by their content hash, and this allows us basically so whenever we build a new version, the built pipeline uploads or manages all the new content to s3 and then build a container image that just has an engine X and the index dot HTML in it, and this allows us to do our rolling upgrades of the UI server, which just says index, but also allows us with our same rolling, where the same back end tools for rolling, upgrade and rolling downgrade in a rolling back when we hit a problem, all we have to do is Rob back the container image to a diff to an older version and we get the older version and then it starts fetching.

A

The older assets from s3 is great, unless you know Esterly Goes Down, which never goes down.

A

Well, yes, so this is actually yeah. Good point: thanks per month it is actually we're number one failure or build pipeline. Put the wrong fingerprint in or one of the common failure modes here is the build pipeline builds the Jess, you know, decides it's got this fingerprint and then, for some strange reason decides to build it again before it uploads it and gives it a different fingerprint anyway. So that's like number one failure. That's really the lump you first.

A

So how did we do that so I'm, sorry I'm going to show JavaScript at conferences we go. This is what it looks like, and everyone in the audience is probably like breathing a huge like water they've done, but we we've got a service on the backend, which is the thing I'll talk about later, which accepts the format. So if you go to slash metrics you'll see text that looks kind of like this. That's there does that fever, so there should be one there.

A

So I sent these around last night and the list of feedback I got on the improper use of naming for Julius was like. Oh man, I had to go fix all the naming this morning, so we sent Ani that looks like the Prometheus Exposition format to our little gateway.

A

Now this piece of code has to be embedded in the index: dot HTML, which is served from our nginx running on our cluster and not in the JavaScript asset that served on s3, because if it is in the JavaScript asset, the server I see we'd never be able to know if it failed to like that. So this is why this is special case. This is special code directly in the index.html and what's more, we set it as a timer and then in the in the asset that you download from s3 we've canceled that top.

A

So this is how we make sure this doesn't fire all the bloody time. Okay, so that's you know there is a to the exhibition, for you should use that in it. So that's that's case number one here, cancelling number one when the this is a react component. Okay, look: doesn't it yes, so this? This is the thing that can't after five seconds, so that says on the slides three ways.

B

A

A

Because it doesn't work, okay and there's that actually you'll see there's a higher rate to the conference. Wi-Fi. All right now so case number two describes the react. Components render something, and so we want to track when they render. When that render fails. So we so dude Jordan has come along and built a little client library in JavaScript for Prometheus.

A

Now he did try and use the existing, no Jess ones, but they're heavily tied to like server-side operation, I've been told so he had to write down and for this basic the little serializer for the expedition format and there's you know we just post that the API minutes. That's basically what it looks like we call this. We call this function when we want to only want to write in there and then- and the next slide doesn't show this.

A

There is a higher-order react class I assume that, like a high order function, but for the class there's used to wrap the other view, so in Phillips talked previously who's here for fills talk: okay, no 101, persecuted by dick concepts of HTTP middleware on the backend, which are functions that effectively wrap your HTTP handlers and when you call them they measure it and report it to Prometheus. A high order class is effectively the same wraps your views and reports that data to Prometheus when they succeed or fail.

A

So that's that case number three high request agency. This code is about code, guess how to cook yeah. She really wrote it last night. So basically we tried to model our JavaScript library on the prometheus best practices and on the structure of other jar of other Prometheus language, libraries, so getting close. But it is a bit fresh. You basically create a and latency metric and that you call it to to record the I'm just like in any other language. So this is here.

A

U88 start. Why is it only doing it already state for David.

C

A

That they'll catch errors and success it'll go if a response was received, okay, good I, think and then in the background.

A

This is where, like I gave up on the JavaScript and just wrote some notes in the background that put is the client pre aggregates these on the client-side in the web browser and it pushes these for the server component, every 15 seconds, I, think and then the counters that you know we set a counter for how many loads we had there not with answers in the traditional Prometheus that, because, after every push that counter is reset to zero, so they're really more like gauges that are pre aggregated on the counter.

A

We just don't wanna overload the server with lots and lots of pushes. This is all open source. As of last night in we've worked upon, guess and all credit to Jordan and a little bit derogatory I was good and a fantastic, especially liked earnings from right.

A

So let's go and look at the back end, the back end: we've owned one and I. Don't like writing my own things and what you might so? Why did we write our own run well on the first page for the Prometheus push gateway. Is the statement that the push gateway, never forget, series pushed to it and we'll expose them to Prometheus forever. The latter point is especially relevant when multiple instances of a job differentiate their metrics in the push gateway, I, don't wanna, read a full read: excitement yeah via an instance label.

A

So the original idea here was that we'd use the way. We'd have an instance label that the clock IP address, but we've got your huge number of users, but enough users that this would block the cardinality of the metrics in prometheus beyond your on what we need. So, hence we couldn't use the push gateway, but we still wanted to me. It had to be pushed as Prometheus, whether it liked or not. It can't go out to our clients web browsers and scrape them out.

A

So we had to build something. So we built the Prometheus aggregating. So this way the metrics and they don't have a label on them saying what incidents they come from. They get aggregated memory, data, nothing right, we've written to disk and they get served out to a fourth and this one's and this thing gets raped by Prometheus, which then kind of you know does its thing. There are, as I've discovered, actually on a conversation on the Prometheus, the Phoenicia, the Prometheus IRC Channel, I, think the beginning of this week, their artifacts.

A

What other things that basically do the same thing and that's in my opinion, you know too many. So what we need is another things that unifies them or famous xkcd anyway. So the first there's this thing called webdriver, which is a front-end protocol for basically what I just described basic fact.

A

This does Jason Ethan I think anyway, that post Docs, a really good guy works at cloud failure. This is the London Prometheus meter, so I trust. He knows what he's doing I actually want to try using that, because I think, like the clock right at other than ones we've written, then there's this thing called the Prometheus user metrics. This is really new. This chap started about five days ago, but it seems to you know, he's going to help. So that looks pretty good.

A

The thing that julius punches is to was taught by Outbrain, and I did a really good reason why we didn't use it I think but anyway, so there's that you should check now as well. That seems to be the most well established, and then there is a slide by this chap that just put a slide deck where he describes exactly the same problem and what they did solve it I just couldn't find any open source software associated with it.

A

But if you do please, let me know so: yes, there are a bunch of aggregating push gateways so, finally, to close the loop on this story, I talked about wanting alerts off this data and not just you know, relying on not just you know humans, and these are the alerts that we manage to run the things the thing I like about these. So first we alert on the rate of errors like I. Don't care about an absolute percentage of errors.

A

I just want to know if more than say right now is, is a more than about temp application load. So that's the HTTP loads I. Think of failing so that that needs to. We can download it in my opinion, but but first we were on the on the rate of errors, so the percentage that fell and second render- and it works- they'll showed you at the beginning so anyway. So let's go, have a look at that. What it looks like demo.

C

A

Going to have to UM mirror so I can see what I'm doing.

A

Okay, so I'm going to unfortunately use we've cloud or not for me here for this because I don't know how to use Prometheus anymore, but it's basically the same. You get to type in nice, pretty things in here, so I have a list of pre-written ones.

A

Histogram and I'll do that last so we'll do the rate of application, UI application loads. So some great.

A

Application stop there. Let's try. It is UI application load, do over last minute and will do by or directive status yeah they go so just to prove it does work presentation.

A

So someone visited our. This is our dev instance. This isn't our production cluster. We have more traffic than this on prod. Believe me, I hope. Yes, somebody visited our website and which is nice to know. So then, the next one we're going to rendered page fails some rapes, UI UI UI page render 1m.

A

Could you do by path? I, don't know what passes in this context.

A

C

Why does the read in.

A

B

A

Do we do funny things with the URL, so the URL is that you are oh yeah. Let here go up there. The URL you see is not the URL. The server serves, apparently this front-end stuffs pretty good.

A

So he go what's what's happening here. Someone has visited from org ID notebook, notebook ID, so are this is caught?

A

You can't see that Kenny, sorry, it's quad off, so this is quite neat and the front-end taps have done the right thing and they've got rid of the organization organization ID and the notebook ID for us to minimize the cardinality of the metrics, with its serving, which is good, which is like and there's no arrows, which is the main things or maybe, let's do it bypassed comma, actually, just it sort of on, though, with no errors.

A

Oh, No, okay red is success.

A

Of course thank God, no one's colorblind right and then this one is my favorite one. There's nothing on the sleeves there's. Actually 20s I, don't how many of you have typed in a histogram query into the expression browser?

A

Okay, not many, and how many of you have typed it in correctly. First time no, you have in it right. So this is why I love the work that David's done, because it makes it incredibly easy to type a histogram in. So what is it UI bucket and then we want to do one uh some rate that and then we're going to need some time there to do this. Do this? Did you fix the bug where yeah, fantastic, sighs, okay, good, bye, Ellie?

A

Did you there? You go first time, but why is the value 10,000?

A

Okay? Well, we need to do it. There's lots last night for the conference, so ten, thousands a bit high I, suspect you're reporting milliseconds in something that seconds, but even ten seconds is pretty high. What do these down here.

A

Of course, today we annotate the stuff with releases apparently now, but that's got nothing to do this talk. This is our dev environment. All the new stuff just gets continually deployed to it and then, when we decide it's good week, we promote order fingers.

B

A

Thank you great, so that is in fact all I have to say on this topic. That is how we do it. Thank you very much. Any questions.

A

Start questions is that a yawn or a crack, okay, I'm, sorry, okay, is like chap in front of. You has a question I.

A

Mean there's only Chrome right: does anyone use anything other than Chrome? What do you use Firefox? Is that still about okay.

A

Yeah, no, it's a good point yeah. So the nice thing about the product we're building is that you guys are our end users. So we can be a little bit more fast and loose with browser compatibility. Aren't we.

A

So the question for the answer the David gave you should be giving this talk for you David and is that the client library is pure JavaScript, vanilla, JavaScript, good flavor.

A

Yeah, so the question was why: why did we pick 15 seconds and why not? Why not quicker, because mobile users are I, don't stay around for very long I think the answer that is mean everything's 15 seconds, like us great period, is 15 seconds. 15 seconds is just what we picked and we should probably push more more frequently. That's a good point. Do we if they browse away before 15 seconds, do we try and get a push in before they browse away? Can you do that?

A

We could do that, but we don't do that. Okay, thank you. Chapter back.

A

That's good idea.

A

Is up so the question was.

A

You, of course, here.

C

A

Well, at least the metrics for the last call, you don't know the ones for this call yeah, but yeah like I'm. You could always stick them in a header. You can control headers from description.

B

A

Up here, David cool.

B

It's this one see. We only do this, it's really simple here, because we only track the start time of the request and the end time, and this is really what counts at this point, because you really want to just have you. We just want to track the experience from when the request was made to whenever when it came back and this probably care how long the server took to respond, because what we want to measure is the end-to-end time basically, but.

A

You could you could take this value and put it on the next query. As.

C

C

B

Yeah, oh for the other way back. Oh yeah I see yeah I like that idea. Yeah, that's good yeah!.

A

So not be done by tomorrow morning minute oh yeah,.

B

Yeah I I tell Jordan it's in.

A

San Francisco, so I thought I might be done here.

C

B

That's true yeah, probably local storage, but no.

C

A

Dancer included unit, you have a job apparently, where cuz we're hiring.

B

It's even better the.

A

One of the things on the piggyback thing I really want to do that with tracing as well, because we do we do no tracing that consists of.

C

A

A

Yeah the point.

A

Can you get only approach, busting of them yeah? Do you have see web the improbable guys server, building that our friends, my cool? Thank you, those release, any other questions, observations. No one's asked the obvious one disappointed is. How is this authenticated.

B

A

Isn't yeah so this is how can you like authenticate this stuff so like Google Analytics doesn't authenticate it? Basically, what we just implemented is Google Analytics, just the thought. Don't review zip line. Thank you very much.