Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 13 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Monitor My Socks: Using Prometheus in a Polyglot Open Source Microservices

Description

Monitor My Socks: Using Prometheus in a Polyglot Open Source Microservices Reference Architecture [I] - Phil Winder, Winder Research/Container Solution

This presentation describes how Prometheus was integrated into a polyglot microservices application. We will use the "Sock Shop", a cloud-native reference microservices architecture to demonstrate some of the best practices and pitfalls of attempting to unify monitoring in real life. Attendees will be able to use this application as a reference point, or as a real life starting point for their own applications.

Specifically, we will cover:
- Integrating Prometheus in Java/Go/Node.js/Haskell
- Best practices: what to monitor and how to structure the monitoring code
- Pitfalls: what goes wrong and why
- Demonstrations and descriptions how attendees can try it for themselves

About Phil Winder
Phil Winder is a multi-disciplinary freelance architect working towards the research and development of cutting-edge technology. Most recently he has been developing cloud-based full-stack microservice systems for a range of clients but has a significant past in machine learning and electronics. His company, WinderResearch.com, is collaborating with Container Solutions, a container consultancy, to provide R&D services in the container space. Phil has Ph.D. and Masters degrees from the University of Hull, UK in Electronics, with a focus on embedded signal processing.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hello, everyone welcome to the basement. I hope you manage to get all the way down here without too much inconvenience. This talk today is called Monica my socks and it's a bit of a play on words. Hopefully what I'm going to talk about is a lot of Prometheus talks.

A

Talk about like a specific implementing from youth uses, specific way like a specific language or a specific tool, or something like that I'm trying to order the scope a little bit and I'm, not just about like this I'm, also going to talk about how you actually deploy it as well, but the slot is only 30 minutes, so I'm going to race through it a patient of not my name, is going down a freelance engineer, my company's called window research, but I've been mainly working with content solution and we works. So how do?

A

How do we start? Well? Container solutions was asked to help out. We work to develop one of their new products or a few of their new products, and this product is called weave cloud and it's a hosted service to help visualize and monitor and integrate with your CI pipeline.

A

So we were asked to help out with this and specifically for for this conference. Part of that service is a service called cortex and effectively it's a a hosted, managed multi-tenant version of Promethea, and it's really cool because well, one of the biggest features is that is backed by comes an s3, so you could you've pretty much got unlimited data retention, speed out the drop dates or any time that's one of the big feature set, but we're also adding lots of added extras that hope to improve the user experience as well.

A

Well, whilst we were doing all of these projects, we we started to realize that we a needed something to test as part of our development, lifecycle and B. We need something we can demo and so we started asking around, you know, will of what can a demo? What can I show- and you know we started- asking even more basic questions, more special questions, but how to do Micra services and there's loads of great literature around there's loads of great blog posts and books and stuff that have been published, but there weren't really any realistic example.

A

You could just get up to speed with straight away, so with that, given what we decided to do was in then the sub shop and the stop shop is an open source reference, microservices architecture and what were what I'm hoping today is that I'm going to take you to take you through a brief tour of some of its that some of its implementation. But the beauty is that you can go away right now and deploy this yourself and look at the code yourself. It's a fully working application.

A

The idea is that it's supposed to replicate an e-commerce website and the way that it's laid out is supposed to be more realistic than some of the. You know. The examples that you see in blog post, I think that the the actual layout of the example looks a bit like this. Each of these circles circles is a service, and these services are backed by certain databases and we've got a few.

A

Other technologies in there, like using stars on the front end, is serving the dual purposes like it's hosting this static content in the web site stuff, and it's also acting like an API gateway to the rest of the services as well. The rest of the services represent different parts of the imaginary sock shop, so we've got a car catalog or the processing line. Things like that and we intentionally made it polyglot.

A

So one of the decisions really early almost has uses many languages languages as we could, and that was so that when new people came to the project, I wanted to try the project they'd be able to use their own background to to get interested in the project with a specific service that they were interested.

A

So we've got services written in, go, we've got reserves, that's written in Java and we've got a service written in know, chess as well, so hopefully it's covering both for an end of backing guy we're also using a range of different supporting technologies and databases as well, and so this is it and we'll skip to the live version. Hopefully so yeah like I'm fine. So I'm just going to take you on a very, very brief tour of what it is.

A

So we've got a catalog here that will give you you can add our items to the cart. We've got like a user service where we can log-in. Ok, password was just password. If you were wondering and then we can proceed to checkout place in order and so that, in a very short sequence of steps, kind of exercise, all of the different services that are going on in the background and later on, I'll show you some of the monitoring that are included in those services.

A

So we can see this process in in action and you can see how many times I practice. This presentation by the number of orders, I've got there, and this is the link I'll put this up at the end. But hopefully you can go away today in try for yourself and later than we think and yeah have a search around.

A

It's there's a short link, get to iOS stop shop or, if you just search for micro services demo, you get there and you'll find this website, and one of the coolest things about the project is that we intend we initially develop the project test, various components against different deployments. So, even though this is a Cuban SS conference, you know if you're interested in visa or nomads offs or any the docket you know product, then you can actually deploy the soft shop 20 of those different deployments.

A

There's like a light for light comparison using the same same service, okay, so onto the meat course, so I'm going to actually delve a little bit into the code. Can I just ask who was here for the previous talk who to talk about data?

A

Okay, it's not opening I will okay, so he went into detail how to do this. In Java we went over all that generate JVM, metrics and stuff like that, but I'll just I'll cover my bit again because I think people will all benefit from it. So the Prometheus libraries there are several officially supported. Prometheus libraries go Java, Scala, Python Ruby. There's also lots of unofficial ones, which, from what I've seen a generally of good quality as well.

A

We use in ojf one in our application and that's solid, so I wouldn't worry too much about using a third-party library. Some of the libraries have helper frameworks. So if you're, specifically, if you're using something like Java, then you might I would double check the the Prometheus project yourself and other people as well, because generally people have already done what you're trying to do if you're trying to get like export geometrics, for example, there's already exporters out there. So don't try and rewrite like yourself.

A

This helped us for first for spring and Jessie and the things like a grenade which will automatically export Ivano related, metrics and opportun stuff like that book first, because this is a bit more of a go friendly conference than Java, then we're going to we're going to look at gopher. The project has three Co services. The payment using catalog services are all written in go, and the one I'm going to look at right now is the catalog is what I'm simplest.

A

It's backed by a database of maestro database that stores the data I'm going to look a little bit of a code now. So this is the first time you might be seeing Amit.

A

So as there's a good question, how many people here have actually define the metric before do to understand what legends about half about how ok so I'm going to go I'm going to take it a little bit slow over this first step, but then I'll I'll be faster on the next ones, because they're all very someone, so here we're defining the global variable called hey, see we listened see and we're using the standard, Prometheus library to create a new histogram and before we're going to further.

A

This is probably one of the most important this so previous support for type of symmetric, a counter, a gauge summary and a histogram. So a counter is a simple count of things. The gauge is like a real-time view of a measurement. Summary is like a summary statistic, and histogram is a histogram of values. So it's like a you know. Is it like a histogram chart that counts in his individual minutes? We've chosen histogram because of a chat call Tom will key down in this corner.

A

Here he upset a few people by trying to define zone a metric scheme, and it came up with this nice acronym red metrics and the idea there is that with red metrics, you can generally monitor the things that are most important to your application and they are the request rates. The error rate and the duration of that call is happening. So the request rate is account. The error rate is the count of the number of errors and the duration. The latency can be measured by a histogram Bukit.

A

You don't want to use a summary statistic like a rolling average there, because it doesn't represent outliers very well, it doesn't represent the range very long, and the beauty about the histogram vector is. Is that the you can count? You can obtain all of these metrics just with the one single history parametric. So with the histogram metrics, you get a count for free and you can do the error, egrant labels, and you also get the book it's obviously in as well.

A

So all of the services that are not to show you all are implemented with a scrum.

A

Ok, now that we've got that out of the way the next most important thing is the name there's some good documentation on the Prometheus website and there's also some good blog posts that tell you how to pick good names to your label or your names, and the reason for doing that is because it's not you the developer that will be using these metrics. It's going to be other people as well than the other developers. Other ops guys could be marketing people stuff like that.

A

So you need a good name that explains what the variable is and they so they can find it easily and so generally I'd kind of summarize all of the recommendations into two things: one is to use SI units, so second bytes meters grams stuff like that and and secondly, be consistent in your naming. So if you've got, if you are measuring the same type of thing like we are going to be here, we're measuring an HTTP call be consistent in what you call it.

A

It doesn't matter so much what you call it well, I mean it goes, but I could have called it like HTTP request generation or something well, it's more important to be consistent with whatever you do choose, because then it becomes much yes to do some more advanced querying and filtering on the on the final metrics. When you get them a little bit, help get that bit look. It is quite important, depending on what you're doing so the histogram said is.

A

It is account in particular bins and you actually have to define those bin ranges in advance, so by default it gives you can't remember thing: it's like 10 10 bin, ranging from 1 millisecond up to 10 seconds, if you're trying to monitor something. That's outside of that range or you know it's significantly skewed to one side of that range, then you'd want to override that and provide your own bin inside it, and the second most important thing is the label, because with the labels, this allows us to do this.

A

So with this little bit of code, I'm defining four labels here, the first one is the method. So this is relating to http metric, so the method is like get post put, delete. We've got the route, which is the path to the HTTP resource that were accessing, and then we've got status code. So, with a request rate, we can just look at the method and route for the error rate. We can look at the method in the route but filter for particular status codes.

A

That's what gives us our error definition and finally, we can get this duration from there. There's two grammar start and that's it really so, once you've defined your metrics, you need to register that metric and you might notice that I said use a global variable and we generally use global variables because by definition, Prometheus dictates that you're not allowed to have duplicate metric names.

A

You can only have one metric for one name, so by definition you might as well make a global and it Sherlock, because I found like during tests and things like that, if you don't make them global, sometimes they can get instantiated twice and you'll get a nasty, Star, Trek.

A

Okay, we've got a quick, middleware definition here, so we can intercept the HTTP call and then we're creating something called an instrument here and in this instrument we've got our metric that we're going to use to write our metric and then we've got another thing, which is quite important called a route matcher and generally, the goal of this piece of code is to try and match routes who they're defined that we put so, if I give you an example on the catalogue, usually we're accessing the catalogue with something like a path of like slash, catalog, slash, ID, to get a particular item in that catalog.

A

We don't want to create a new metric for every single ID, because we could end up with thousands or millions of metrics and that's a very quick way to degrade the performance of your Prometheus.

A

So what we do is we match that route to a predefined name, so it'll probably look like it'll either be given a name if you're using like gorilla, mocks or something, but in other languages we maybe just replaced the ID with a fixed string like : ID, or something like that and then we've finally gotten the thing that does the monitoring and it's a bit long. But it's actually quite simple: we've got a bit in the middle here. This served HTTP. This is this is actually calling the method that actually performs.

A

Whatever the end point is doing actually does what the end point doing, and then we've got like pre and post code in a pre code, we're starting a timer in the post code or calculating the G and then there's a few little bits around it, and then the last line here we're actually doing the writing of that metric. That we've just measured we're passing in the method, the route, the status call and a WebSocket thing is don't worry about.

A

But these two things are carefully from the code above and then we're writing to the metric with the observe method and passing in the number of seconds that it took.

A

That's kind of it really. This is just a bit of wiring click to to pull it all together. This is kind of general standard. Go code, go code, we've got our endpoints for our particular services or creating our route earth and all the HTTP calls we've got we're defining the middleware in this instrument.

A

Thing I told you about so this is where we're passing in our metric, our global metric and that's creating the middleware, and you can have you know more middle words and log business just like that, merging them all together and wrapping them around the root term, and that's it we'll go.

A

So, let's have a look at something else. Now look at some Java code and I will go through this. Despite the previous talk, which is more like more in-depth Java, metrics and monitoring. So if you're interested in Jeffster difficulty take a look at Alexander's talks before me in the the Marcoses demo, there are three Java services again and we're going to look at the car. I, say: generative, just a digression slightly just to vent my anger sha for Java, because my I've got a young daughter.

A

Two daughters and the eldest she was really into looking at these flashcards and on the flashcard they had je the jellyfish and so I had a massive argument over my wife like why y jc-- jealous she's, two years old, she lives in England she's, never ever going to see a jellyfish until she goes to the aquarium when she's 10. So why not J for jellyfish J for jar, something different, so I took method into my own hands and now Jays a java. So if you have a daughter now, J is for Java.

A

So if she becomes an engineer later on and blaming this, this is this is where it all started.

A

Within the Java services, we chose to use spring Boop. We did this because it made it very easy to get up and running and make a microservices in a very short amount of time. Some of you use it great. Some of you love it. Some of you hate it. Various reasons generally spring boot is all controlled by like aspect-oriented programming, stuff, so sort like annotations, but if you're just using core Java or something else like the Java EE or something most of these, things can be done by extending classes rather than using annotations.

A

But we start really simply because we using the spring boot helper that comes with the prometheus library and it comes with enable prometheus endpoint and that single line will handle all the backend stuff to expose the slash the metrics endpoint and set up all of the Prometheus registers and stuff. So very nice, little little caveat, though, by default spring boot comes with something called actuator.

A

Actuator is a spring booty thing to expose loads of metrics and I've got a little bit of code to disable that that it exposes on slash metrics by default and I'm, changing the prometheus endpoint to be slash, metrics, so I'm, basically just replacing them. So the couple of options for doing the timing. The easiest one is to use another annotation, there's a Prometheus time method and that's great. However, this by default uses a summary type.

A

A summary type is something that I haven't gone into detail, but basically it's a summary statistic: it's a rolling average measurement. You can provide quantile to compute on those values, but the problem is that if you had multiple services and try to you, if you had multiple services, you can't aggregate a summary type, because those quantiles have already been calculated. So that's bad and also, if they're, not very flexible. If you change your mind, if you don't want to measure the 99th percentile and what changes in 98 or something so I wouldn't use that method.

A

If I were you, what I would probably do is write your own annotation to do what you want, probably with a histogram. If you're doing this type of stuff and I'm surprised it's not in the library- and it probably will be soon familiar and you can see, I've been very helpful and provided you with all the code that you need to do that, but what I did was I went with convention with the other services and I wanted to monitor all of the endpoints.

A

So I did some spring jiggery-pokery to insert some interceptors, which are like, like middleware, basically and I added my interceptor to a particular group which was just all routes, just a glob there.

A

Now we've come to the actual definition of the histogram and you'll notice that it's very, very, very similar to go. Implementation and you'll find all of the libraries follow the same pattern. You know you create a new thing and you're passing it names and label names and stuff, like that, all very, very similar.

A

The Interceptor that I'm creating has has got a pre handle like a pre hook and it's got a post handle post up and if she's doing the same thing as the go code is doing where we're starting they're starting a timer in the pre hook and then were calculating the duration in the post up pretty nice.

A

So I would probably recommend that you just write your own annotation, though, especially if you're using spring or spring loop, creating an annotation that would create histogram with your specific options and that's probably easy to use in a lot one final, one nodejs and express nuts front end. Obviously rose through this one, we're using Express the Express framework and from client library, which is one of the unofficial libraries, but it's very very similar to the others, but creating a global global variable called the request.

A

Duration seconds again same name: it's got the same labels, yes pretty much. Similar Express has got a middleware concept, and this is implementing that for that middleware in effect and same thing again, we're starting a timer calculate the duration there and then writing it.

A

Expose the metrics function at the end, throw some way and that's all well and good and knowing you've got the code, but what I've seen previously is that not many people talk about deploying that, and since this is a bit, this is a. This is a kubernetes conference. I thought I will try and show you deploy that a little bit.

A

A

Normally, when you finish your code, you get some sort of meme and get like this during flack, and this is my entire mean quota in one slide. By the way you took people, so great you've done the codes now, if it and it's actually not that simple I would say that writing the manifest the kubernetes manifest and getting them all set up and set up in the right way can actually be as difficult, if not more difficult than writing the code yourself. So I'm going to take a look. Quick, look at that.

A

Remember that we can deploy to other orchestrators, but I think someone would probably get pissed off if I started. Talking about me just for something's to execute it and I assume you're all fairly familiar with manifest. If you not just go to the repo, when in fact you can go to the repo, irrespective of your experience and and look at all of these in detail, but basically we're creating manifests that contain the Prometheus image.

A

Passing all of the arguments to the Prometheus binary and the configuration file is being passed in by a volume man and the volume mount. The Prometheus song channel is actually a config map and inside the config map we have all of the Prometheus scrape settings, and this this is really where the the chunk of the work is to get Prometheus set up. How you want it to get set up the one in the repository. This one is just the same as the example that Prometheus give you to it to monitor.

A

Communities works really well at the box. If you wanted to do something more advanced, that's where you'd start and then finally, we've got a service Prometheus service, it's exporting at a port. For us, you can try this out yourself with mini goose and it's as simple as doing that and you'll have a full, fully deployed set up, basically start mini coop I'd recommend opening the memory. There's a lot of services there, this sort of 810 services and plus all the kubernetes stuff, as well I'd, put two to four gigs of RAM.

A

If you can and then clone that repository, do cubes ETL create on the complete demo file and then a cube CTL create on Manifest monitoring folder in there. Wait because it takes a while to download all of that and then, if the demo gods are smiling down on me, you will get.

A

Something that looks like that a so. This is a version of the same thing, that's running in the cloud not on my laptop because it would grind to a halt, but you can see: we've got lots, different names, races there, we've got specific name safe places for the sub shop. We've also got the namespace the monitoring, and that's where that Prometheus deployment has been deployed.

A

Interestingly, someone's booked, the number of front-ends I was that you, okay, thanks for that yeah and if we I've got an SSH tunnel open and it's forward in Prometheus for me and if we would type in generations seconds. Oh, where did that go.

A

Then we should see our variables there. There you go so all of those services are now being scraped automatically by the the Prometheus configuration and all of the services are exporting metrics on slash, metrics, awesome.

A

So once you've got Prometheus deployed what you do next, the next thing to do is start thinking about what you would like to do as a engineer or as a business. But we've gone through some examples with the stop shop here. So the first one is for Sonic. So kevanna is a dashboarding tool, often comes with Prometheus, and it's very cool I'll go through all the details. We can have a look in the repo.

A

Basically, we've got the graph, our net pendency and service, like just like we do prefer Prometheus and the coolest bit is that we've got one massive massive conflict map with the Jason graph and dashboards inside. So all of the dashboard to start and get in you know, version control and they get pulled in and then written to Gravano when it's done, there's a little batch job to ingest and write the dashboard to the configuration file and then there's a few extra bits to get some more information about cuban areas and disk usage to stuff.

A

That is what it looks like. Let me see it's. The live version is running ooh almost so that doesn't look great, but here we go so this is live dashboard of of what's going on at the moment, or it would be a life dashboard. If we can see some metrics go on.

A

A

Damn you Wi-Fi I might have to go back to the picture back up, sir. Oh, oh.

A

It's getting there.

A

Too slow too slow too slow, too slow, okay, but we've got dashboards for performance. Ironically, we've got dashboard bombs and we've got default dashboards for the error rate, the request rate, the latency of the service and things like that and an error rates and stuff that's very cool. This is a dashboard specific to the sock shop. Resources is also kubernetes resources as well. I just want to trim CPU usage and RAM usage and things like that and then regardez very simple, analytics kind of dashboard.

A

Let me see if we can get that help Ashley, because that's a bit of a smaller dashboard to load, and then we can see how many socks we've sold. 37,000 orders have been placed since this start. This this example side run and so clear, I'll be a rich man when this presentation ends once you've got the monitoring in place. The next step is alerting and alerting with resources. To make a little side.

A

Project called alert manager happens is Prometheus sends its alerts, which could come from multiple sources into an alert manager and an aloud manager handles all of the deduplication and all of the aggregation and then spurts an alert out to one of many services. So again, we've got manifests in that repository that define the alert manager, dependency and again we're using a volume mount to pass in the settings file for the alert manager and inside the settings file.

A

We've got lots of things like your a slack secret and all of the other options and which define how you want alert manager to alert, and in this case we have a kubernetes secret setup, called a flat hook, URL and that's being written by this little script into into the configuration file. You also have another config map which defines your levels and in this case it's an example, alert rule for a high error rate.

A

So if we've got, if you look, that's our metric that we defined in our code, if we've got a status code at 500 for any of our services of greater than 1 for 5 minutes, then we're going to get an alert and slacks and I'm not going to try and do that, because the Wi-Fi will fail on me I suspect.

A

But what it looks like is that so that's a very knotty example, but you know in actual usage, we've got alerts, are all sorts of different things or any anything from service health, all the way down to performance- and you know user, metrics and stuff like that.

A

So to summarize, what I would say is go and check out this project on those URL and try and deploy it yourself have a play about with it break it fix it. If you like it and want to contribute back, then, just again, it's on the repo check out weave cloud and specifically the cortex part. We cloud it's a really cool and a you know: well developed sophisticated version of Prometheus, which is more in telling multi-tenant and hosted there's improved all the time. As well happy pill all there, thank you very much.

A

Any questions please, I.

A

I couldn't comment on that um thread: I haven't got enough experience with link a t2 to common and there's probably about people that can answer that question. So I would forward you to some of my colleagues that were better able to answer that question.

A

So yeah see me afterwards and I'll point you in the right direction, if you're interested in knowing the answer to that.

A

A

Well, check out the alert manager project I mean that's where all the documentation is, but, as you can see, it's actually remarkably simple. It's just got a simple syntax, specifying alert you've got if a Prometheus query: okay, one like having somebody else to talk about.

A

Yeah I think I can't personally not in this toxis. It's not to talk about alert manager, I have seen other. There are other talks about alert manager in particular, which you could probably get more detailed information there, and you know quite honestly, it's going to be down to you eventually I'm sure, there's patterns you can copy, but it you know, everybody's alert is different, preferably everybody's business. You might be more important in you might be more interested in certain things or other people, for example, again sorry useless.

A

Asked the Prometheus developers next I mean personally personally: I would choose it, because it's super simple and I love that you know all of the other, especially proprietary systems, are very complex and that's hard to understand and therefore hard to use and how to deploy and how to manage various incomplete comparison in complete opposite of. That is super simple everything is well described and it's basically very hard to get wrong and it does the job. So that's all I can ask of a monitoring tool. Okay.

A

So when you work around your manager, like that end, point I think is coming in and what other things turn out like requested to do up? You might want that. Yeah yeah sure you can just yeah, monitor all all activity and Martha know all of the examples in the stop shop for all incoming the only the only real, the more complex example we've got in there is that we've got like an orders pipeline, so you've got one service that calls another that cause another cause on the left.

A

So all of those do report metrics and then the top-level want the orders. One is kind of an aggregation of all of our that's the most complicated situation we've got in there, but yeah I can imagine that you would like to measure things going helpful. um This is a monitor on the incoming plane, but also microfiber.

A

Compare right. Is there a problem between yeah yeah and you never know where the problem? Yes from a neighbor community? Yes, yep! Okay! Thank you very much.