Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 13 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ops for Developers - Monitor Your Java Application with Prometheus [I] - Alexander Schwartz

Description

Ops for Developers - Monitor Your Java Application with Prometheus [I] - Alexander Schwartz, msg systems ag

Prometheus is ready-to-go to monitor your infrastructure and your containers. Capturing simple metrics from a Java application can be done automatically. But Developers need to go one step further to provide useful application and business metrics. This talk shows how to collect meaningful metrics in Vert.x and Spring based Java applications. It applies Dropwizard Metrics to gather internal application statistics, and Hystrix to provide metrics of the external interfaces the application consumes. It also shows how to fill gaps by writing your own collector. By providing end-to-end examples this talk enables developers to make their application ready for production.

About Alexander Schwartz
Alexander Schwartz is Principal IT Consultant at msg systems. He’s been in Web development for more than 15 years and enjoys productive working environments, agile projects and automated tests. At conferences and user group meetings he talks about the things he is passionate about.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Welcome to the talk box for developers, monitoring your driver applications with Prometheus, just a short survey, the two usually two kinds of people, one who arrives java application, the other one try to run it so who's trying to run java applications all right who's. Then developing java application, this room alright. So there was a people who do both that's good right good, but then, let's see how we can do this and to help premiere. These can help with this prometheus. Well help lots of people already heard talk about what pro music can do for you.

A

Then I talk you through the setup optic of today's samples that I show you and then we see some how to how to monitor java applications. Central it's all about for today, I'm a I'm, an RT consultant I.

A

Do Java programming also have a Java t-shirt with all these cool people around here, she's a bit strange, but I develop IT software for Java soccer for more than 15 years and on other programming languages like PL sequin or if the databases quite different and well I've, been working in consumer finance, doing like online credit applications and online banking, so web applications. Well, it's a second home and well marriage to kids and well hobby known, then, what only known so an awful little add breaking again I'm doing them prometheus monitoring retreat.

A

So if you're free on either the 23rd and 24th of true and living near, the Frankfurt area may like pre-register, we not yet sure about venue and the date is going to be either finder and feathering or contact me after the talk so Prometheus. Well, what we want to do we do. We want to monitor some house and applications, and we want to send out some loads and the thing that we use to solve all this. At least for today. It's going to be Prometheus and.

A

As we all know, prometheus is an opinionated solution. Julius gave a great target prompt from last year and well it can do a lot of things: instrumentation, collection, storage, but well today we are looking at instrumentation. How can I get all this information out of my java application into Prometheus and then maybe do some dashboarding but I'm, not very good at dashboards anyway, all right so to set up for today Prometheus the thing in the middle.

A

How would you then, like any other application, you can do you see advisor to monitor an application when you write it in the container you can see from the outside of CPU usage, the RAM usage, but well, that's not very much if you want to kill our virtual machine, but it's a start, and you probably should continue doing that if you monitoring the availability of course service. If it's running as your users expected to run like doing probes and probably also do the alerting you can do something use something like a black box exporter.

A

Please continue to do so because the metrics we will see today they're mostly for well analyzing problems and type people into that, but they are not really for alerting. Other thing use something into end and find out what your thing are doing.

A

There's also dry mix exporter I will not cover that today, because well with Java developers, or you at least probably know the Java developer. You are running the application, for he can put something into your in the into the application that you're running and there's going to be the simple client well so forget about the rest of the of the other three. For today we will concentrating on the simple client well I've been told.

A

It's been called simple client because it has been a not so temperature before, let's see what we can do with the simple club and well, we want to do some cash growth and how things look like if all the demos work really so well, let's start with something well quite familiar: I hope start with C advisor.

A

If you want to do some RAM monitoring, we have a variable, that's being exposed container memory, usage and bytes, and then we can do all the namings and find out the right soccer or containers that we want to look at, and we can do all these from cool expressions here. We also can do the CPU use it through container and it's as we all know, it's then measuring in seconds used and yeah. That's where we start. That's the outside Java Virtual Machine, it's from the outside, like any other application as well.

A

So let's, let's take it some click again. Some little deeper s use a simple client.

A

So there's like we looked at the driver at the RAM use it from outside. You can have two ram usage invites you can then have a metric wife used, and then it gives you information about all the RAM tools that are within a java virtual machine, so the heap space and the other spaces that are around and it also breaks down for you, the CPU usage from the outside. You can't we see if your application is doing something meaningful or if it's just fully doing a garbage collection.

A

We because you're used to wrong Java memory parameters on that and well, let's see how things look for that. Let's see if microfauna dashboard works so from the outside. Yes, you have these containers and give you from the outside. But let's look at the RAM usage.

A

So what you see here, you have a like. You probably know from the go program. We've seen it as a talk before you have the heat used heap space, it's always going up and down, and if you put some load on that application like let's use a B for that.

A

You will see that it's going up and down, but the cpu use it for for for the garbage collection is still done quite low.

A

Put that into the criminal run it, and then we should see it that it's behaving a bit differently, all right and up to now the garbage collection was very idle. Every time there was a heap compaction, we had a small spike of 0.4, nothing and the moment we do some proper work. We also see that the garbage collection is doing some more work, but still we see that it's doing proper work like delivering responses, but our memory cueing is then probably quite ok.

A

So that's the first insight that we can get from here and what we can do for that. We there's another drop library. Let's assume we're running in a spinning, good environment. Here we just say default X, Falls initialize.

A

That will include a JVM metrics here and we will choose an URL that we want to expose this metrics on I like it to be after manage, because all the spring boot applications behave like that as it's just a simple servlet, you can also use it with any Java EE application or drop wizard or whatever it trusts the servlets that you put in there. It's runs in the Java, Virtual Machine and well.

A

It takes a memory takes in a library that you package into your java application, but it gives you the incisor to be seen and.

A

So and then, as soon as I browse to that URL with all the metrics, we will see in a minute that there are lots and lots of other metrics as well. At least one starting with JVM.

A

Object diversion info as well I get tomcat into here, because I'm running in a better Tomcats the threads I'm using. So it's getting a lot of information here and it helps me to analyze problems that arise, but not it's not really useful for alerting, maybe the garbage collection, CPU usage. That might be something you could try to do alerting on what all the rest is just for information and digging down when performance is low.

A

So these are the simple metrics that there are for the java virtual machine. But then you probably also want to go ahead and set some proper business metrics you want to have you have special rest call or you have a special business function in your code that you want to collect timings on and how loftiness is it's being called? What you can do there?

A

You already have to draw a simple line in there and then you add probably that metric to the screen boot- that's quite well integrate into spring and you can translate a drop Wizard metrics to the Java simple lines and then all the information flows through committees. We still have this one URL understand not only providing a JVM metrics, but also the drop wizard metrics. That include your information here and what I did here there is now I have to enable metrics so I little bit of tweaking for my spring good application. Here.

A

It has some more proxies here that you probably have anyway and extend another class okay, that's too much of typing, but then at all the very places around your application, where you want to gather some timing to put an annotation here and it gathers the epic timings I, give it a name. The counter call example: this is doing some proper berkeley and, in the end, press returns. Ok, the business code stays the same, but at this very point I get now all the timers that are around and.

A

It gives me, then, a metric that starts with the name that I put into my driver called the counted call example. It gives me some timings here is 75% percentile and 92 98 percentile, so yeah so I know about how how long this takes on average and so yeah. Let's do some load testing on that as well.

A

I'm. Please do ask questions if you want.

A

Okay, so you you're asking your in Java EE what to recommend.

A

While the driver Dre BM metrics that are easy, that they will just come out, I.

A

Thought that I should look that up yeah a dry mix exporter could help, but you want to get tying some special metrics on special places around that you could do an intercept. Well, I have to look it up, so I'm not prepared for Javy, not anymore. Sorry, I used to do some lots of Java stuff, but I'm currently doing spring. That's why I'm prepared to spring.

A

So I run the test here.

A

And I go back to my my dashboard to many browsers in this local area and I now get the dashboard where I can see.

A

The latency of of the calls and I can get myself a dashboard on that one, and also the number of let's see how many other metrics we get around here. Counted calls.

A

A

And this shows you also the documentation that there is or basically not is because all these metrics are converted from drop wizard, we're kind of losing a documentation that you had their account here from the name. Ike I know what it is, but it doesn't give you all the other help information that you usually get from a proper Prometheus exported.

A

So if concept snapshots one minute rate, all the information I can call to my dashboards all right.

A

I, don't know if you've ever come across circuit breakers. So if you built an application and you call another service or you want to make sure that if you call an external interface that you in your own application, have a proper time out that it's turns within which the decor returns within a reasonable time with the timeout, if you also have bulkheads orchids, meaning that you don't trigger too many parallel requests and if you limit it to ten, the eleventh request will return with a failure to don't overload the other service.

A

If it's just slow what you can do here, there's another driver, annotation here called hystrix command.

A

So this annotation just works with spring but--. But if you do it manually without an annotation, you can make usage of history in java environment as well, and so it gives you lots of well information here like how many times get you call that exact external interface. How many of these calls work successful? How many exceptions have been thrown?

A

What's the latency with all the percentiles that we've just seen, and then you can do this this rate and put it on on dashboard, so I haven't have to pair that as well. So let's take this test here.

A

Put some dough to my soda.

A

And this now gives a district airport with all with both the counts and the lighting scenes in one diagram. I, don't know if it's really useful, but it's helpful for this talk is two x's one on the left, one on the right and when you now have this diagram, you can either somebody who is interested how this the case is performing in production. You can find out if your application slow, because your peers are slow or are you slow because well, the database is slow or you're just doing too many confusions in Raleigh.

A

So, and this gives you a good idea how things are performing with your external interfaces, both in terms of number of calls, as also in the number and the latency that you're getting from that external interface. Let's see what else we get here, so history.

A

Gives you well also the information if a circuit breaker is open or closed so circuit, breaker being when too many errors occur, it will switch off that interface and then you want to monitor that and probably alert on that as well, that a an external interface has been cut off automatically. The good thing is that it will we end able the medically after given amount of time yeah. It also uses thread, pools and semaphore, so you can monitor here as well and also the different states of what can go wrong.

A

How many, how many four legs have been triggered for the various commands that you have? They will be visible in here and gives you metrics in your dashboards again well this. Actually, these are now proper, Prometheus metrics, so they come all with an proper health text, and this is the SoundCloud history exporter. But there is just it's a pity that has not yet been released on maven central so that, but you can just compile it yourself, maybe even put it into your project as a module and work with it.

A

Hopefully we will soon have it in simple exporter. As a module that you can put it in and use it like any other exporter as well, hopefully our time over the next weekend to do a pull request on on the original project.

A

So, but there is a one thing in here: let's call B where there's no there there are histograms here, I can have these % house, but there are no Prometheus style histograms who has worked with Prometheus stars histograms before not too many all right. The thing is this gives you the latency of one application instance running and at some point you want to aggregate. These Layton sees over several instances of your application and you can't really, if it's already calculated to these 90%, we can't really aggravate that over several nodes and real Prometheus.

A

Histogram will then deliver the raw data to Prometheus, so you can do proper calculations over list of nodes and then detect outliers a bit more about doing so then. But then, if you just want to have this the metric set of form for a single node, you can register the district's Prometheus metrics publisher and put an annotation on every external or every method that is wrapping an external service call.

A

And then you have all these timeouts and bulkheads and the metrics that you can export and if you're not using spring, you can do a wrapper class for that proper up ourselves and don't need an annotation and we'll do the same for you.

A

A

All right, this is going to be the next slide. I have a sample for that. um Hopefully, in the next few weeks, be a pull request for that, because I think they just give it a rough start and just exported metrics that hystrix has inside for deciding about.

A

So this race is collecting the metrics inside anyway and did one, but we want to run exporting these metrics to Prometheus, but they are not not meant to be aggregated, like Prometheus once they're to do so. If you want to do real histogram here, so yes, you had this annotation, so that history is enabled and this histogram.

A

It looks like this line here down here, like you, do a histogram quintile for 95% and then sum it up over some nodes and force for the commands that you're interested in and that's the real Prometheus histogram that you want to use and the adapter that they called admitting content is missing today and that will there be that will be there in future.

A

Hopefully you can hook into history and that's the lower part here and hook into the stream that tells you about every executed command of history and with once each of these has completed for various reasons. You just update your histogram and then you it will be probably exported like today. You have to do it yourself, maybe yeah. Maybe somebody contacts me afterwards and we do it together. I, don't know I'm happy to share what I've done so far.

A

So let's see and then the proper that the garbage collects and the proper.

A

Histogram export looks like this, where you have like less or equal than that timing. In this case, you have that number of calls and the longer you get the more calls fall into these there's culminated packet kind of, and you have the total number of calls that is usually equal to that course, and you have also the the total XE to execution time, and this is a real basic gram that you can extract on that. Then you can aggregate over several commands over several nodes, the same command and or compare one node with other nodes.

A

How they perform is that external interface and like the things you like to do with Prometheus.

A

And also do I have to turn yesterday when the current Java JVM exporter doesn't or only exports the CPU ii issues for garbage collection. It doesn't one it doesn't group in that in a histogram like that- and I yeah and this this code is also prepared on the laptop locally and there going to be a pull request that you get a histogram for the garbage collection. So we can find the outliers and yeah and.

A

You're welcome, and so it looks then like this, that you see how many garbage collections have been done and then you can do well, whatever you're doing with so Prometheus a so right.

A

So we are now been diving into how to ride your own collectors and, let's see a vm garbage collection metrics. This is how that code looks like. Let's see if it fits on the slide. You you extend the class, define the labels that you usually want to have on that, for example, garbage collection, that's the X from the course and the garbage collection type you're running on and the buckets that are the default buckets are just put in here, for we are to have them ready and read for the buckets.

A

Are you construct a histogram here and once there is a garbage collection you subscribe to that event like we've seen with hystrix ones and yeah, and then you collect them at the very end and you're kind of done and.

A

Don't so if you not using any annotations- and you want to do native Prometheus metrics- that's the way you can do that it's simple classes that you had to need to trigger counters now and then interceptors would be nicer, but it depends on your environment, enlighted memory, all right, so that's real histograms I was a bit surprised that I didn't get proper statistics for spring but-- regarding databases, so I wrote my own tomcod statistics collector for the ftp things than the andy databases.

A

So when it comes to Tomcat, you want to know how many threads are used. How many connections are busy? How many connections are open, an idle almost the same with the database? But then again it's a driver class like we'd, like we've, seen for the for the whole drive en you extend collector class Rider collecting and then you hand over some metrics from the data source. That's the other thing in this case.

A

There's a bit different than the Java Virtual Machine garbage collection, we've seen before there we had the histogram created early and then we updated that all the time here we taking some values out of all the data. So just some counters and housing.

A

Right so well, that's the end of spring boot, I'm, just wondering who's running vertex, right to view right. It's fun right, sometimes some issues, sorry yeah, and sometimes you want to just figure out what's going wrong and which way is going so we're take, there's an embedded in event, bus make you run different verticals on vertex and they have an event loop and they run on different threads and they communicate beta t using in event bus and what you develop is vertex. It's it's an HTTP server.

A

Maybe you can do custom protocols if you want to that's something I'm doing, and you just want to know how many sockets are open, the usual stuff and again, let's do another load testing on this.

A

Oh yeah, here we go so vertex is now getting busy and I want to know all about the HTTP requests. So second and the open networking sockets and if I go into editing here, I have a request rate is just a request. Count I can get from the HTTP server, that's embedded into vertex and the number of open net sockets yeah, just a variable passed on to the export here. So right.

A

So how do we tell vertex to be that grateful to us to give us all this metrics so on there yeah and how things to do during the setup when you start of your vertex instance you're, giving some metrics options and the vertex is been originally designed for droplet metrics, so you pass on a drop wizard, metrics option and registry name and then, when you after you get, then you initialize the default exports and then you pass onto the driver, simple client. That is then converting the drop wizard metrics to Prometheus.

A

You look up in this shared metrics registry, that vertex registry, that you just created up here and you don't and while you're done with collecting all the metrics, you just have to export them in an HTTP server. That's the second step, not in the server but a matrix handler as it's called an in the vertex world, and then you have your metrics for vertex right.

A

So here we sing a sample and yeah other thing. As I said, all the things that we've counted here are about the problem. Another metric I forgot about you can measure the number of messages in the vertex eventbus that are lingering around and not have not yet been processed. So that gives you a bit of the rate of congestion that you have new vertex running here. That might be interesting for some up to that might be useful for alerting and well.

A

The other one thing that might be useful for alerting is the CPU usage by word from the shrink. If that goes high, something is wrong, and should we look at only other things, we've seen so far today, they should be, from my perspective, not be used for alerting just for metrics to dive in and to optimize the system. If you want to contradict them. Please do now.

A

All right so well, I'm a developer and I say at least for me that it works for me: I do lots of load testing and instrument my applications with Prometheus and then monitor the dashboards during no tests, it's team friendly. So if you operating infrastructure, please give every developer team their own Prometheus. So if they can play around this, it's also code and developer friendly. Well, I can put all my configuration in a git repository. That's really now nice.

A

The only thing that I can't put into a grid properly or is a dashboard there's no DSL I can write in my IDE and commit.

A

Great there's a project I didn't know about: okay use the gamma to describe dashboards. Thank you very much so now we can probably do to properly do grab the yellow.

A

Okay, now we can do everything in the gate and and well and I, really really like the simple setup go to mission alert manager that we know. Thank you very much for the ones who do the go programming here. I can't I have to rely on that thanks a lot. That's the end, thanks for listening, something's and the slides are online already.