Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 11 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 360° Monitoring of Your Services [I] - Philipp Krenn, Elastic

Description

360° Monitoring of Your Services [I] - Philipp Krenn, Elastic

"With microservices every outage is like a murder mystery" is a common banter. But it doesn't have to be! This talk gives an overview on how to monitor Spring Boot applications, since they are increasingly popular for building microservices. We dive into:
- System metrics: Keep track of network traffic and system load.
- Application logs: Collect and parse your logs.
- Application metrics: Get the information from Boot's metric and health endpoints and store it.
- Request tracing: Use Sleuth to trace requests through a distributed system.
- Uptime monitoring: Use Heartbeat to ping services and actively monitor their availability.
All the data will be aggregated and visualized in Kibana, giving you an all around view of your system.

About Philipp Krenn
Philipp is part of the infrastructure team and a Developer Advocate at Elastic, spreading the love and knowledge of full-text search, analytics, and real-time data. He is a frequent speaker at conferences and meetups about all things search & analytics, databases, cloud computing, and devops. Philipp lives in Vienna where he enjoys experimenting with software, organizing meetups, and sports.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Let's get started so we're not late for lunch right. Nobody likes that first off smile, so I can prove to my colleagues that I was working today.

A

Here, perfect monitoring, all of your services or micro services or whatever you have. We don't really care about that. Let's jump right into the topic: I work for elastic the company, doing elasticsearch la Cubana beats or the open-source products I'm part of the infrastructure, team and I always say this is a UNIX pipe I kind of pipe that into developer advocacy. So I do lots of talks about all the goodness we do so, let's see what we have here.

A

This was kind of the starting point of this talk, the tweet that, once you start with Mike resurfaces, every outage becomes more like a murder mystery, because it's so nice you have everything so distributed and it's so hard to find what is it even going on and we do have quite some components how to monitor stuff and it's actually quite easy trying to keep the overview of what is going on. So that is kind of the starting point.

A

So the idea is, we need to monitor and log all the things everything we have in our system. So, just a quick show of hands who's using all the latest.

A

All right, I, don't know micro service is highly distributed, whatever hard to monitor yeah. So how do you log in monitor just who is using our sec? All ready and elastic stack? Okay, that's a few yeah, and then there are lots of other options, but I'll show you what you can do kind of with the latest versions. What we have so we we will use a really simple, app I'm just using spring boot, but it doesn't really matter it's just about something, quick and easy to write.

A

So you can do some some rest codes, basically I'm not doing anything complicated, so I would say it's stupidly, simple, I'm, not doing any load balancing or to discovery or anything, because that is not really core to the problem of the logging and monitoring what we do, I'm just showing all the pieces of information kind of we have and how to collect them and how to see what is actually going on so yeah those who haven't used spring boot.

A

Basically, you get a curated list of all the dependencies you can use and then it's just a few lines of code you put in and it does all everything for you. If anybody has quest for that, we can talk about that in the lunch break, but yeah, let's skip the Java code.

A

Now I actually run that all on the clouds and you will need to interact with it because you will generate the traffic that we are monitoring so I'm using AWS light tail, also because it's kind of the hottest cheat on AWS but like it's their cheap virtual private service and it was kind of the easiest to use I'm just using Ubuntu, but it doesn't really matter and yeah that's kind of the intersection we want to monitor and since we also have a cloud offering I'm, storing the elasticsearch and key bonaparte are using our cloud yeah, and that is generate this deck.

A

We won't really use lock. Search today we use beats, are lightweight agents written in DOS, we do have native binaries, they will collect the information forwarded into elasticsearch, which will store the information and then Cabana can yeah. You can query stuff, you can visualize and you can graph it out. We have dashboards. You will see that so kind of the three components is what we will cover may be generally I'm using terraform and ansible in the background.

A

I'll push the code to get up later on, but again it doesn't really matter so we will package something when we deploy it we'll run it, and then we came to the monitoring part. That is the interesting beach. So, let's start off the rest is pretty much a demo. Let's hope that demo gods are merciful today, so I have I call it bad jar, it's just a jar or java application that will consume memory and crash at some point with a lot of memory exception.

A

Some people may say this is a kind of the default behavior of Java yeah. It's like two lines of code, so that is very easy to write. um So, let's simply stop that so I'm setting in X MX of let's say 512, megabytes and I have a jar that I have already uploaded up and it's called bad jar yeah and it's running and it's not doing anything, um but we probably want to keep an overview of what is going on in our system. So this is cabana in version 5 we have lots of pre-built dashboards.

A

One of them is for processes. Luckily, we can search for that and if you type it correctly, you would even find it- and here I have two servers. I'm monitoring, you can see yeah! Ok, we have some number of processes running and here we have. This is this is the CPU usage and this is the memory usage and you can see suddenly something weird is going on here. We have this pink spike here and when we hover over that, we can actually see oh, it's Java.

A

So whatever is going down on the system level of your system, we have metric bit that can collect stuff like top or metrics from applications I'm just relying on on the top information. Basically here so you can see, CPU is yeah I'm using less than 1% here and less than 1% here, not much going on here, but there this yellow this pink spike, is going on and when you don't always want to refresh your page, we can also change that to be Auto, refreshing, I, hope. My process is still running and hasn't out of memory.

A

Yet and when we scroll down here- and you can see- ok, it just crashed, because you really can see the spike here and now my memory usage has gone down again and when I get back to the shell yeah, you see standard Java, behavior I got an out of memory exception, but thanks to give other, you could really see. Ok, something bad happened here and then you can dive in and see like what was actually going on and what was the root cause of that. So that is the metric part.

A

That's not an interesting yet so, let's get started with our application, so I have a very simple job application, which I will just run on port, 80, sooraj or opt, and it's called front end, but actually I should start the monitoring component in parallel, Java jar.

A

Okay, we'll get to see later on and again this is also spring boot application. You can see that it's basically what spring boot does. It is quite nice logging output, you can see syntax highlighting and at some point it will tell you. Yeah process is running. It has been started. It's running on that port, we'll get back to that later on and now I can actually start. My front-end application.

A

And that should start up. Yeah I was in half a minute and at the end it should say: I've I've started myself on port 80 yeah. It looks good burn to port 80, and now we can see what is going on in the network layer. So we have something called packet beat.

A

It's uses the same library as Wireshark, but running Wireshark in the distributed environment is always a pain in the ass. So we kind of do that. For you, we collect the information. We just read the information from the network header. So, for example, we know okay, this is HTTP. This was the request. This was the response. The whole thing took, I, don't know hundred milliseconds. The response code was a 200 and the URL you hit was fubar.

A

So let's open up the dashboard and there shouldn't be any requests right now, so we haven't done any requests and now you can start your interaction as well. So the whole thing is available as Franta sierra dotnet, slash good. That will generate it or should generate 200, and if you run that everybody gets the phone out and you see you will get 200 I hope. So if you do that so front, end thought Ciara, dotnet, /, good, thick Ciara part is by the way I wrote 13 of my last name.

A

So those who were back in the good old days when rot13 was common yeah, that's where that name comes from. So if you hit good, you should have, you should have some traffic and when we switch to that dashboard, you can see. Okay, we've got lots of requests, so we got 59 requests for good. Okay, we had some five icons. Then somebody hit didi yeah. We have a few 400 force.

A

That is also expected, and you can also trigger an error that is called bad and that just gives you the default white label arrow page and in the background I just throw a runtime exception, so it just generates whatever Java exception happens, but it is a 500 if you had to hit that a few times.

A

And auto refresh is on then I assume yeah, haha yeah hi, it's not available. Then you can see okay, we we also got 25 bad queries now and I haven't instrumented my application at all for that I have just run packet beat, and it's really just listening on the network interface and extracts all that information from the headers, so you're not really doing any hard work or instrumenting your replication or server or anything you pretty much get that for free.

A

So you see how many requests you had, what your top URLs were: good bad yeah, fun with flags and you're, not a typical German audience, because you're not doing just what you're being told like good and bad like yeah at previous talks. I had people write me kind of messages here like great talk, she'd talk whatever so so yeah. You can kind of abuse that and to see whatever message you want to have pop up, then we can see. Okay, these are all the error codes we have and down here.

A

We also have a list like these are my are my top URLs and then you can dive in like what are my top URLs or also, where are my error codes and if you suddenly get more 500 or 400 fours on your page, and you don't really know why this might be something you want to dive into so yeah. That is probably a good starting point to to monitor your service. Okay. So we have these five hundreds and it was the slash bad URL.

A

Now we would really like to know what is causing this issue like what is going wrong on my server and you could lock your application like this, and you see that that is the exception. My bad something went wrong. That is the runtime exception. I am throwing, and then you have a long stack trace and you can start collecting that file and then parsing it but parsing everything that this multi-line is always a pain in yes, so we are not doing that I'm always trying to go the easy way.

A

Instead, since elasticsearch is storing Jason and handing out Jason again, I'm actually logging to Jason already so just to you know anything you tab to show you SSH, no, not like that.

A

Net so I'm, just stamping out my logs as Jason log front and no front end of Jason I. Think yes, so what I'm doing is I'm using log back whatever it logger you have.

A

It doesn't really matter as long as you can write out Jason, because then you have it in structured format already and then you don't need to parse back all the information, because if you use these line statements, where you have something, yes, you can parse it back with a regular expression, but I guess very few people actually enjoy writing regular expressions at least I. Don't really so. This is much nicer. You already have the Jason and we can just collect that Jason and then search in that Jason.

A

So if we go back to our browser, we can now start discovering data, so we have data in different indices. One of them is file bit, and this file bit is actually just pointed at that one log file and collect that log file. For us we throw it into elasticsearch, and now we can just start searching through it. So the first thing I would be interested in is what is the distribution of good stuff and bad stuff, or what is the distribution of messages I have? So you see, I have a lot of errors.

A

I had yeah. Since everybody now hit the arrow page. We only have errors up here and infos down there, but normally you would have infos and errors, and everything like wildly mixed together.

A

What you can then do is all we see. We have really a 50/50 split, that is, that is kind of yeah yeah good split there. You could filter, then down on something you have in that message, but will just display one and you can see. Okay, we have the caller like the stack trace. We see where it actually happened. The process ID the message and we could just search for search freely in here, so I could just say bad and okay.

A

We had 76 bad requests now, and it would just find everything that contains this bad in the body. So you can just really start searching in your exceptions. You have the information structure, so you could also limit the search to specific field. You could say Jason, dot, class and then I, don't know I'm just interested in exceptions from a specific class or when, if you log the the IP address the username, the URL, whatever you log, you could have added it here and then just limit down to whatever you're searching for what is the problem?

A

What you do you want to actually find so that is kind of convenient for log management, which is in tail F. For for log management, you should be ashamed. No I'm kidding come over to us. We have some nice tools or many other sucked yeah soon, as you have more than five service using tlf, it's probably painful I, don't know I, at least for me it was okay, so that is up and running now something else we can do. We want to know if my service is available.

A

We have something called heartbeat and it does just continuously ping your services. It can do ICMP, HTTP HTTP stuff, like that, so I have stopped it for now, so it doesn't ping my server all the time for a change. We can actually look into the configuration file because it's so simple heart beat heart, beat game.

A

So what I'm doing is yeah I, don't copy my password, but I'll kill the instance afterwards anyway. So this is actually the cloud instance where we store everything with username password and kind of secret ID, and this is my pinging. So all I'm doing is I'm, saying fronting zero net on port 80. Slash good hit that every ten six seconds with the time out of ten seconds and expect a two hundred back, and if there is not a two hundred coming back, it will be marked as down so I am starting the service now.

A

Again, we have a prebuilt dashboard, so uneasy I didn't build anything myself, I'm relying on my colleagues so we're interested in the heartbeat dashboard.

A

And you can see it is this up up and running so that should be good. You can see yeah, it doesn't doesn't yet have how long it took. But if it has been running for some time it will tell you how long the transcription take. How long did if you have teal TLS? How long did the handshake take? How long was HTTP roundtrip and how long did it take to see connect on the TCP level? Oh so yeah, it's it's appearing. We have like. We don't have lots of data, but you can see okay.

A

The round trip was there really high? It has now fallen quite a bit, so we can let that run. You can see how long was the distribution of HTTP requests, depending on how long they take like that took hundred seconds. That is like forever. Most of them should be down here, so they should be much faster yeah, and if you let that run, you will at some point see okay. How long did my requests take until I get the proper response? Back and now everything is good.

A

Let's kill the service for a change and if everything works,.

A

It should turn red in like five seconds. Yes, and you can see. Okay of the events we had suddenly here you can see the spike oops, it's going red. If you have multiple services, you will have a percentage of all the stuff, you ping and what is available and what is not available.

A

And if you have that up on the dashboard, you can then react to it or you can just put up the response times on the dashboard and see if suddenly, your response times are not where you want them to be, should probably step up. Okay,.

A

So that we don't generate too many requests, there's the wrong window. I will stop my my heartbeat again, so we can focus on something else. I am starting my front-end again, and one of the nice attribute of spring boot is that it automatically exposes some end points where you get as Jason, which is very convenient for us again. You get some information information about the service as Jason, so what we support is.

A

A

Yeah, that is what the health endpoint gives you. So if you don't want to ping it like from an outside service, you can just use this endpoint and we will tell you up or down if something in the app is down. It will tell you yeah this space currently and it has another endpoint which we can currently use.

A

That is metrics and there you have stuff like how much memory is being used, how a garbage collection, doing like all the stuff that Java applications are interested in and we didn't write it, but somebody from the community wrote the beat to actually collect that information and store it into elasticsearch that is called spring beat so for spring boot. There is spring, beat.

A

Let's quickly change to the directory yeah, so that is the github name of the guy who wrote it and the project named spring beat and we can simply run spring beat now and the spring beat is again pretty much configured like the heartbeat configuration it will just every 10 seconds. It will collect the Chasen information and store it into elasticsearch for us, and then we can. There is no default visualization, but we can simply go to discover.

A

We have spring bit here one of the indices of the information we collect and it didn't collect anything yet.

A

Are you running, please be running.

A

Yeah, thank you. That is the common mistake there is. There is no bad in spring bit, unfortunately, or fortunately we can also, let's get rid of that field. First, we don't need you, let's go back to spring a bit and there you see, we have the health information and the request next to it should be stuff like the garbage collection. So what is the heap size you're, using how much memory is available? How many HTTP 200 requests? Have you served all of that information? So you have all of that stored.

A

There is no pre-built visualization for it yet, but you could easily craft it out like build. Some bar chart pie chart whatever based on, for example, on the heap usage, and if you heat hit more than a certain heap usage, you know. Okay, something is may be going wrong with my java. Application, like heap, can always be tricky with those so yeah if you're not using spring boot, but you're exposing your own taste information. It should be pretty much trivial to collect that.

A

So we provide a base library which connects to elasticsearch and just handles that part. You only need to kind of collect the information you need and put it into the right fields, and then we will store it in elastic search and you can graph it out later on. So that should be very easy. So, whatever jason information your explication exposes, you can very easily collect that and then stored it.

A

Okay and then we want to see what traces we have. So, if you want to see that yourself, but you don't have to it's on a mobile phone- is probably a pain. So, let's see let's go to 1330. So, let's see in the last hour all the traces we had.

A

This is Zipkin, so there is sleuth the detective that is part of the spring boot application, and whenever you get Qwest into your system it will attach a unique ID to that request and that, even if you call remote services and everything that ID is still kept on, that one request, and we can then collect that information and based on the ID, we know what kind of where did the request go and how long did it take?

A

And then we have Zipkin keenest visualization part for that, and so you can see most of our requests have taken very little time like we had microseconds when we caught something there's just the visualization part.

A

So somebody tried to call, and no that was that's not where did that request come from that is interesting.

A

Let's limit it down to good.

A

Okay, that is, that is a regular request. It stripped out the URL, but it knows we had somebody hit good and it took I. Don't know 150 milliseconds for the entire thing to get around, but that's not that interesting. It's just a single call what we can do.

A

Instead, if you still have your phone open, you can do call slash call, and that is actually triggering one request internally and then triggering another rest call in the background and I have put a random sleep in there of up to two seconds, so you can actually see that something is slow in your system like when you called it a few times. It will take some time and yeah.

A

It will always print out hello world, and sometimes the call is slow, and sometimes the call is fast, and that is another very nice attribute of distributed systems. That is very hard to know like something is slow, but what is actually slow? What slowing down my requests? So, if I and you have done some requests now, we know that we should have called somewhere in there.

A

No, not cool I probably need to find traces again and we can switch that back to all, because those will be the slowest anyway and you suddenly see okay ooh something took 900 milliseconds. That is really slow. Let's see what is going on here and then you see: okay, somebody something called call which took 902 milliseconds and then it's called further home which took two milliseconds. So obviously the asleep or the slow path was on this leash call and the call behind it, which was slash home.

A

That was actually fast and if you have lots of cascading calls, you would have all the calls down here and you could see graphically like how long is each one of them taking. So it is once you have added that to your application. It is very easy to pinpoint like what is causing the slowness system, otherwise, with the distributed system, it's very hard to see where the slowness is actually happening.

A

Okay- and that was pretty much the demo so just to quickly elaborate on that- what you will have is I have enabled that in springboard spring booth has an integration for that, and you will get logs like that. So first something is the first, the core thing you've called this calling home and this you called home so that those are the two methods you're calling and then we have three elements in there.

A

So first off I've named my applications. You have called the front-end application. Then we have the trace ID. That is always the same over one call. So one once you do the call or the request against the slash call that ID will always be the same for all cascading calls behind the scenes, but that span ID will be different. That span.

A

Id is basically one unit of work, so it's kind of one method called further and based on that, you can see which one of them is actually the slow one and then we're sending that to SIPC in which is the server to collect the information, and it supports an elastic search back-end as well. So you keep all the data in the elastic search, and it has this visualization part where you can actually see these cascading calls yeah. That was sip King and there is a single, simple, sip King package.

A

You can get from spring boot and there's nothing more to do. It's just another char. You run so stuff. I would love to see like what I did. That talked something that would super be or super. Nice is if we had annotations in Cabana, where you could actually annotate like I deployed something here, and then you see ah suddenly my five hundreds go up or all my response times go up or down. That is super helpful.

A

If we had that there is an open issue, not sure when we will get that something else, I would love to see is correctly for our cloud. We don't have an API, so I always need to create an instance manually but yeah as soon as we have an API. We want to integrate it in to terraform again, so everything can be automated yeah. What are you missing but come like talks, pretty short, come to me afterwards.

A

If there is anything you have think like, that would be great at the year what we don't support at the moment, we are always sitting interested always interested in ideas around that yeah. To conclude, we've seen yeah, we had very simple setup. We had just three instances in the cloud we saw system metrics where we collected basically the top information like CPU usage memory with the network packets. We monitored the uptime of services by explicitly pinging something and seeing how long does the service take to respond?

A

We collected the application logs which we dealt out to Jason because it's way easier to collect afterwards we have the application metrics. That is mainly what the spring boot metrics and health endpoints are exposing. So you can collect that and you can easily do request tracing if you have some trace. That is the same over all the calls.

A

Yeah I think we have like five minutes. Less left I do have stickers. I need to get to knock them out. So if you want to have elastic stickers, come to me afterwards as well, are there any questions.

A

Everybody wants to go for lunch. You can see it. It's! Okay, yeah! If you have questions, come to me, I'm, happy to answer them thanks.