Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: AIOps: Anomaly Detection with Prometheus and Istio - Marcel Hild, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

AIOps: Anomaly Detection with Prometheus and Istio - Marcel Hild, Red Hat

As IT operations become more agile and complex, at the same time the need to enhance operational efficiency and intelligence grows. Monitoring applications and kubernetes clusters with Prometheus has become quite common. Yet identifying relevant metrics and thresholds for your setup is getting harder. In this talk, Marcel will show the tooling used to collect and store metrics gathered by Prometheus for the long term. Then analyze those on a large scale for extracting trends and seasonality but also forecasting of expected values for a given metric. Finally, he will integrate the predicted metrics back into the Prometheus monitoring and alerting stack to enable dynamic thresholding and anomaly detection. All done with no more than OpenSource tooling and a fully working demo utilizing the instrumentation available in Istio.

https://sched.co/Nrvj

A

Okay, so we're starting right on time, 1120 I'm, a German person, so I like to be really sharp thanks for coming to a talk on AI ops, I'm, really glad that so many people made it here and as I saw a talk today on ml ops and as we use the term AI + ml interchangeably nowadays. So if you talk to a sales person, you probably talk about AI.

A

If you want to get money, if you talk to a data scientists, you talk about machine learning and if you talk to an engineer, you probably talk about an SQL query, so the term AI ops means how can we use AI to help operations versus the term ml ops is rather used for how can we operate machine learning in an operational environment?

A

So this talk is about anomaly detection with Prometheus. Just a quick question. How many of you folks in the room would consider yourself an operation sky, maybe you're a chance, and how many would consider yourself as an engineer software developer one and how many a I'm data scientists, that's good, ok, great, so lesson learned should be: don't be afraid of AI and ml. We're trying to make things easy.

A

Now, click it. It doesn't work anymore.

B

A

Okay, so I'm. This is how I look out like in on the internet on github I'm, a software engineer, slash manager, hacker zombie, slayer I'm from northern Germany, a city called keel which is close to Hamburg and Denmark. We do a lot of shipping. Only people not containers, so we're not cloud native, yet we're still mode one I'm working for a stealth, startup called Red Hat. This is our brand new logo check it out. It's on the internet. It's really new!

A

You can get some swag down there at the at the booth and I'm working in the office of the CTO, also known as octo, which is like an octopus.

A

We get to play with a lot of cool stuff like making things sure things, don't explode or implode, but in the essence, were trying to define the gravity of the next technology. That's redheads looking into or trying to invest. So it looks dangerous, but it's actually a lot of fun and just to give you a little bit of understanding what my team is working on is we're trying to understand how reddit see say I.

A

So, first we want to make sure that a I as a work load of machine learning as a work load runs very good on redheads products. So we want to make sure that if you're running your workloads on open shift say, then it should be a really good experience, as you would do it on your local machine. And then we have a project called pop, which is taken from the expense series.

A

That's the thing where all the evil scientists work and those guys are looking into making AI stacks really good and give you recommendation on the high stakes and software sex in general and they're also recompiling tens of specifically for your environment and we able, just by recompiling it to squeeze like 10% out of the tensorflow project and as a side note. I love talks with a lot of pointers and links so that you can do something after the talk, so I spread all those small post-its on my slides.

A

You can take pictures during the talk or at the end there will be a summary slide with all the post-its on it.

A

Then we have the open data up, which is another great project, because you know data is the foundation. Data is the new oil people say software has become so ubiquitous stone. So now people are looking how to make. How do you use everything out of data and we started the project called the open data hub on open data, a biome? It's a one-click solution to create a data science platform in your kubernetes cluster.

A

So if you go to the operator hub and click on the button, you can install the open data up in your cluster, which will give you steps F for storing, which will give you a spark for analyzing stuff, which would give you give you a tuber to help for you notebook requirements. It's really easy and you could try these things out. That I'm, showing here in this talk on the on your local, install or on your cluster at home, and then we concentrate on a I powered products and services.

A

So, for example, we have a service called reddit insights which use some machine learning technology to identify problems in your data center, or we are met trying to make openshift really stable by looking into the metrics that the clusters provide and clusters provide metrics. Nowadays we have Prometheus, so we want to try to solve it. Trying internally to have the openshift team make better decisions if their development, their cluster rollout, is going well. So this is what this talk is about.

A

How can we look into Prometheus metrics a little bit more intelligently, we'll look at Prometheus. What prometheus is? How do we set up some long-term storage and to analyze your data? Then we look at the definition of an anomaly and I will look into how we could integrate it into your existing monitoring setup.

A

So a quick word of notice to set expectation. This talk will not give you a shiny product and the holy grail of monitoring. It will also not give you a ready solution that you can just plug into your monitoring setup and turn it into this old-school spider demon from doom. You also don't get a success story, how we created our or transformed our messy setup into a really advanced AI monitoring solution, but I'll give you the tools and the scripts to get you started.

A

We will look into some questions and answers to problems that you might face in your journey and the good thing it's all open source. So you will find everything that we do on the internet, which nowadays is the equivalent to get up if you're talking in code. So what's prometheus, who knows what prometheus is in this cross? Okay, keep your hands open up and who played with Prometheus and quite a bunch and who uses Prometheus in your in a production, environment, cool so but then to get everybody on the same page.

A

Here's a quick architecture, diagram of how the Prometheus setup looks like and everybody loves architecture, slides right, they're so easy to decipher. Let's start simple, so we have this Prometheus guy and as we're in the kubernetes worlds, we have to name it from Greek gods right so Prometheus is the guy. That's brought fire back to the people, so hence this flame and then we have targets, and these are the things that Prometheus monitors or wants to monitor and those targets targets expose metrics just via HTTP endpoints, really simple.

A

The metrics are the current state of the application or the target that you want to monitor. So that's really important to understand, because we can't go back in time and ask the application. How was your state five minutes ago, but it's Prometheus, adding that timestamp to the monitoring data that you want to scrape it gets tricky if you're running in a disconnected environment like in a your cloud provider which might give you the data back in a time slice interval.

A

So you want to make sure that your data scientists understand that domain know that problem and then Prometheus stores the metrics in its time series database, which is essentially really performant database for this kind of data, and you can query the TS DB with a powerful query. Language called prompt QL, which is kind of weird to understand if you're coming from an SQL background, so it will be, you will be dealing with vectors and some multiplications, but there are tutorials all of the internet to get you started and now metrics don't mean nothing.

A

If you don't get notified, so Prometheus can store rules which will trigger targets and then it'll push out words to a thing called alert manager was also part of the distribution, and this guy will take care that you're being notified.

A

So in essence, Prometheus is made in its core for monitoring and alerting built around a very powerful and capable time series database. So what do we need for machine learning? Anybody data exactly so the giant had once data. Let's give him some data before we give him data.

A

We all obviously want to give him enough data, because giant had once really a lot of data and for me yes, we'll only store it for like a day two days and then throw it away because it's made for monitoring it's not made for storing it for long term. So we set out like so this project is like a year old and it's I would show a bit of the progression here.

A

We set out to look into thing called panels which I thought was also a Greek god, but my kids corrected me he's from some Marvel movie so, but he looks really powerful, so he can. He can store the blobs in an object storage, so he basically takes that TS, DB and stores it in s3, SEF or whatever, and then it has the Prometheus API on top of it.

A

So in essence it would give you unlimited retention until you're running out of storage, and you can query the hosts historical data right via the prom QL and Prometheus API, so it's pretty transparent and it also can downsample your data. So if you have have it on a minute scale, but you only want to retain it on a 5 minute or 10 minutes Kay. So it can also run these jobs in the background. But at that's at that time it didn't work out so well in our open ships. Cluster, maybe was openshift.

A

Maybe was tunnels, I don't know. So we look into something else like in flux, TB. It's also great thing to talk to store time series database, a time series data because in essence in flux, is also time series database. It has also nice integration with Prometheus, because you can just configure a remote, endpoint and Prometheus will send every sample over to in flux, store it there.

A

But the unfortunate thing is it really eats Ram for breakfast, so it tries to keep everything in memory to keep everything fast and we ended up with like 6 12 cakes of RAM after just a day, so it wasn't really really suited for our use case. On the other hand, you can sew your data scientists log, because you can connect your pandas dataframe, which is a tooling that the data scientists work in rights to in flux.

A

So they will have a really easy way to get into the in flux database where, as Prometheus, they would need to learn how to create Prometheus.

A

You could solve this problem if you would buy an influx cluster license and scale out, but unfortunately they have the open core model. So they will only give you one node for free, which is okay. So if you want to use in flux or ya for in flux already in your vironment, you can easily use that.

A

So we created a Prometheus scraper part that was just query. The Prometheus API store the JSON blobs in some safe storage and some s3 storage. Put it away and store it. There, storage was cheap and we just had it backed up them. We wrapped it into a container configured. We an environment variable and pointed to a Prometheus instance. Job done the good thing about it.

A

You don't have to talk to your monitoring, guys to configure their monitoring setup to move their data to somewhere else which they might not want, but we just need access to their Prometheus store and we just kept the data out, which is pretty straightforward, and then you can use something like Apache spark to query those JSON files and, as I mentioned, it's part of the open data app. So you would get that for free in your notebook, and this is an a screenshot of such a notebook where your data scientists would work in.

A

He just points it to the s3 endpoint and then with SPARQL. It feels like a database, so you will get your pandas dataframe. You can create with query it with SPARQL, which really feels like SQL over all those thousands and millions of JSON files, and they just feel like a database which is great, oh, and it also has like some integrated snice function like you can create the standard deviation of a large time series in a clustered environment which would not be possible in a single node environment.

A

So it's Apache spark is really good for processing large data sets and doing ETL.

A

Then we revisited tano's a couple of weeks months ago, and we now have it actually running in a production setup. So all the OpenShift for clusters that you install will send back a subset of their telemetry Prometheus data to to Red Hat and were capable of storing like 360,000 metrics per hour, and we store it off into a Stano store for long term.

A

So there's a blog post about it, how we set it up and it's all open source that zotero's works great now and I think it's a de facto thing how to store Prometheus data for long term. So what do we really need for machine learning, not just data, but we need consistent data.

A

So we need to understand how the data looks like there's a rough saying in the data science community that 80% of your time, you will spend on the nature of your data and just 20% on writing your neural networks and doing your actual AI stuff. So a lot of cleansing goes into this, so we needed to to translate what the monitoring guys know about Prometheus metrics to our data scientists. So let's look at the Prometheus metrics metric ties.

A

We have a gauge which is essentially a time series and we have a counter which is a monotonically increasing time series. We have a histogram which is a cumulative histogram of values, and then we have a summary which is slightly different, so a gauge goes up and down a counter. Just goes up easy, histogram a little bit harder. So in a typical histogram, you would bucket your data into slices and count how many values fall into one bucket, so I have 100 in the 0 to 5 bucket.

A

A summary is slightly different because it not only counts how many things fall into that bucket, but also give you concrete value out of this bucket. So if you want to know, oh, my latency is actually 499 in that in that bucket and not in a histogram. You would only know that it's from 400 to 1000, so you get an actual value out of it. A metric in essence and I think in to speed it up a bit.

A

A metric is in essence a series of data points which consists of labels and a measurement so value and the timestamp. So we would have cubelet operations as the metric name and then you get it some give it some labels like hostname and operation time, the operation type, and that is one time series in Prometheus. So you have to be very careful, careful choosing your well labels so use something that is finished in your labels. Right.

A

If you put some infinite data and your labels, you might have a component combinatorially explosion and you're just using them wrongly wrong because they're using creating a lot of time, serieses and then you have like the values. Okay, so here's an example what he would store in these values.

A

So monitoring is basically hard. Prometheus, for example, doesn't enforce a schema, so slash metrics. The import can expose anything at once and as the slash metrics endpoints is provided by your application, and your developer decides oh I'm gonna change that metric, then you're monitoring guys don't get updated because there's no schema there, so their alerts don't trigger anymore. They think everything is good, but the Deborah developer just changed his metric name. So it's it's hard and then we have a lot of metrics.

A

So in a typical kubernetes cluster kubernetes cluster would see no Dex for and some of the services being monitored. You end up with thousand metrics, which is lots to understand, so the state of art in monitoring right now, I think, it's fair to say, say it's dashboarding and alerting, and the dashboards are created by the subject matter: experts which understand their stuff and they create dashboards and do some learning. There are no tools like to explore the metadata in metrics and that's what we tried to create and the tools where you start from Jupiter notebooks.

A

So here are some examples of these notebooks. That's live in github that you can run point it to your data, run the notebook again and you get the same results pretty straightforward.

A

So looking at the metadata, like the labels of your metrics, would be the first step like how many metadata is actually there. So here's a screenshot of something where we see what is it like the month over five months we see the unique labels plotted and you see that it started out with fewer labels and then continues to have a little bit more labels, etc. Here's another way to display the label. So you see a clustering few of those we used a technique called ste distributed.

A

Stochastic neighbourhood encoding, which I can barely pronounce oh I, mean I practice it a little bit and you're, but you don't necessarily need to understand it because there's a notebook, you can just run it and you get this nice blot. And then you see some smaller classes and you can ask yourself as the monitoring guy, why are there some smaller classes of flavors? What's what's what's wrong, they're looking at anomaly types, if we want to define an anomaly in a time series, we need to understand the components of the time series.

A

So it time series might have a trend, so it might go up or down. It might have some seasonality like it goes in the morning. It fluctuates and then the evening it goes down and it's the same every day. And then you do some crazy shopping event and it goes really really high up. So that's the seasonality that you could extract from those metrics or it might have an irregularity. So there's a sudden spike and basically an anomaly anomaly is something where the time series doesn't look like. We expected it to look like.

A

So we might look at the seasonality. Oh that's going up and down, but now it just goes down. So that would be an anomaly or yeah. It could have a seasonal anomaly.

A

We looked at the tool called profits from Facebook, it's still maintained and it's a pretty pretty nifty Python library soon pointed to your time series. The black dots is the actual data and it will project or predict the data, how it would look like in the future. So it gives you an upper value. It gives you gives you a lower value, so that's a bound bounded window where your values should should land in and it gives you yeah the actual value where things that should be, and it also predicts the trend in the upper picture.

A

So in this case it goes up and in the lower picture you see that the daily seasonality of your time series it's just pretty straightforward and pretty easy to use.

A

Then you could pretty straightforward, say: oh I'm come looking at my value and compared to the actual value, and if it's not the same, then I create an alert but I think you don't want to do that because maybe it's just an outlier, so we also looked into different ways how to actually determine that. It's that it's an and the easiest thing would be an accumulator approach where you just count up a counter.

A

If you see such a deviation and you increase the counter- and if you don't see it, you decrease the counter again, but on a higher level. So you subtract two and if your counter goes above a certain threshold, you would call it the anomaly and here's also some tweaking and playing in game in place all right. So let's look at our architecture set up so far we have open shift or the kubernetes cluster. We have you at the application. We have familiars monitoring your application. Then we store the data and sell for internals.

A

We have a jupiter notebook for the data scientists to play with. We have SPARC installs guess it looks a little bit too complicated to do it in half a day and we're in the kubernetes world. Now it's all a one-liner which you pipe into a bash script and you have a massive set up already, so we kind of expect that as well from these tools, so you want to play right, so we wrapped it into a obviously into a container. So the pink is I. Think yeah.

A

The pink box is the thing that you can readily deploy. It would have the forecaster profit model there and some other models in there. It would store its forecasts in a PBS and then we export the predictions. We have the same way that we do our actual monitoring, so we just provide an Prometheus endpoint. We expose the slash metrics thing for our values, so you can hook it right up to your Prometheus instance or another Prometheus instance that scrapes this normally detector.

A

It's all up on get up in our AIC OE org. It has a taco file, so you can build your container yourself. It has a openshift built pipeline or it's on Kauai. Oh, so you don't need to even build it. You would configure it.

A

We are a environment. Variable pointed to your metric name that you want to do the anomaly detection on and down. There are the predictions you can set up, some alerting rules to be fired. If this and normal is the anomaly set to one and let your monitoring guys look into it so demo time everybody loves demos.

A

Let me see how I get out of hmm okay, so I don't have a mirrored here, so I have to so here's my open shift cluster were installed Prometheus and the training part, which is doing the anomaly detection and the forecasting.

A

Yes, these slash metrics endpoint that is being exposed by this part, and you see here if we zoom in a little bit, Wu predicted node load one for year y hat upper okay, so that's seems to be our prediction can go to the Prometheus UI.

A

And it has this nice, nice graph, but that's just a bad UI for doing some initial explorations, so you want to have a craft Hana dashboards where you actually can visualize it so also. The graph on the dashboard is also something that comes with this deployment. So here on the upper graph, we see the prophet's model, how it predicts the upper and the lower bound and the actual value the load of one of those of the pots.

A

No of the note in this cluster I see it's going up and down pretty nicely and it also retrains them all. So these points years when we retrain the model because it needs to look back so you might have a online training model which adjusts like online all the time on incoming data prophet doesn't work that way, but it needs to look back and then does its computation etc, and the red line is the yeah I said it's the actual loads, and here we have a spike which seems to be weird.

A

Let's see, that's another prediction by Fourier analysis. Fourier analysis is much better at prediction: action predicting actual time seriousness. You can see it as the blue lines match up quite nicely with the actual value, unless not here, so there seems to be something wrong.

A

Unfortunately, prophets didn't detect this, so the anomaly detector inside for prophets they didn't detect it as an anomaly and I would need to look close. So maybe I need to adjust some thresholds there, but for real detected as an anomaly, which is good.

A

So this setup you would get just with a deployment channel for your kubernetes cluster and you can play with it. Maybe no load one, it's not the best thing to play with, but you would work with some of your application developers or you look at projected disk usage or CPU usage pick something that you are familiar with and start playing. I. Think that's the lesson learned and here just to show off the nice cool thing. That's the open data website. Let's go back to the presentation, go into and I'll go into full screen again.

A

So we made pig head happy, he likes what yeah and he can go away. So here's the picture with all the links to those notebooks, etc and, as you are all pointing the camera to me, may I also point the camera to you and make a nice selfie for the folks at home. Everybody cheer you like the talk. Yes, alright.

A

That's it question time: let's back up starts I have five minutes for questions, which is great.

C

So, are you still using your custom, storage or, or is this now just calling the tonneaus query layer to do all of its computation, we're.

A

Now using so we're applying that to an actual internal team, the top team which deployed a craft database- and they don't know how to operate it, because it's there's no no kubernetes deployment there.

A

So it's exposing some metrics, but they don't want to create alerts for it and we store it in our ton of spec ends, because we have all our Prometheus instances set up with tano's as a sidecar, so it automatically stores at long-term, and we also have a Python library that can query eight, honest or Prometheus back-end for the data scientists with some notebooks so that they can just create a panda's data frame out of the Prometheus or channels set up just by some two lines of Python.

A

So that's that's! Our internal set up. It's pretty straightforward!.

C

B

On this thanks, a lot for sharing and I do have. A quick question is that from a monitoring system that sometimes when we have a normal, like we have like two hours, we're crazy and system goes very high. How do you just to rule out these abnormal? Because after these, like two hours period of normal and this part of dirty data, we call it was just a fad to the predict of next data points? How do you just rule out in a real-time system, yeah.

A

So if you have two hours that are where the system is crazy, yeah and you expect it to be crazy, because you know that your users always check the website after they come from lunch, then the model would see it because your training train it on a month worth of data or a year worth of data. They the model would expect it as a normal behavior.

A

So it wouldn't trigger an alert if you're saying that it's not normal in these two hours, then I hope the model would fire an alert and if you, the behavior of your folks change- and they don't check the website after they are coming from lunch, then you need to either retrain the model which is done here on an hourly basis, but you might want to configure it to a larger timeframe. Obviously yeah your modeler, just we haven't looked into online training. So that's that's something that we don't do yet in this set up.

A

We have another prototype, called the Flatliners. Also on that organization which uses reactive X, then the Python framework, which does some online computation but I, wouldn't call it AI and machine learning. It's just some statistical analysis so far, but there's a set up to do streaming. Data analysis with Prometheus data.

A

The mic is coming I.

D

Am what issues, if any, did you have with Prometheus monitoring? What what are the challenges? What were the challenges so.

A

The the first challenge was to get data out of Prometheus, but I think that's solved now for our set up, but he will face the same challenge in your setup because giving a sight card to your Prometheus to the monitoring guys is something that they would have objected to and you need to store your data somewhere and then the other challenge is definitely understanding the problem domain, understanding the nature of Prometheus, because it's another, it's another thing to learn for a data scientist, so that took a lot of time, understanding from ql and then also getting the buy-in from the subject matter.

A

Experts because they think they have their monitoring under control and I think, to certainly create a habit under control but actually convincing them that what we do can provide some value to them.

A

It's also not so straightforward, so we're still struggling internally finding stakeholders and finding users of this this stuff that we're building, but then we're in the center of in the CTO office. So that's what we're used to and always meeting edge and that's what we like.

D

Thank you just a so the the way you got data out of it is there's a Panos provide syncs.

A

Tano's is just stop storing the data away, so it takes Prometheus flop. Time series puts it into s3 or safe sources, they're done, and then you query tano's and knots Prometheus over all those flops in the s3 bucket, okay, okay, I'm timed out. Sorry guys! If you have questions come here or I'll, probably be at the reddit booth.