.Net Foundation ML.NET, 2 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Anomaly Detection Transform in ML.NET

Description

Using a ML.NET transform to detect anomalies in time series data.

Code - https://github.com/jwood803/MLNetExamples/blob/master/MLNetExamples/AnomalyDetection/Program.cs

Contact:
Twitter: https://twitter.com/JWood/
Blog: https://jonwood.co/

A

Me everyone, so if you're not familiar with Tom series data, it is simply data that is captured at a specific town, such as a company's stock price, is a classic example of Tom series data and here's an example of Apple stock expressed as Tom series hearing. So in this graph the closed prices on the y-axis and in the date is on the it's pretty much just a line graph and Tom series analysis is mostly used for a financial or IOT applications.

A

However, there's often so much data that it's impossible to go through at all manually to see if any of the data points or irregular or anomalies- and this is where anomaly detection comes in and in this video I'll show how to do it with an ml Dannette transform I'm here in Visual Studio and our to have a dotnet core project console project loaded here and first, let's take a look at our data and, as you see, is simply just a column with dates and in a column of especially our energy consumption.

A

On that that date and Tom- and so the first thing we need to do- is to get our new get packages downloaded so first, we need to do Microsoft. The ml done net package in version 1.2 for this by the way.

A

So after that, we also need the Tom Series package as well.

A

All right, what does loaded avoid we'll go back here to our program and like always, first thing we need to do is create a new email context.

A

Then we need to load in our data, so I do it contacts a data load from text file and we need to create kind of a data schema class. So we'll do that I! Just do it down here.

A

Call the energy data.

A

And here I want to tell a load column, zero.

A

That's going to be a date/time and we just call it date.

A

For our next one, who called one.

A

It's gonna be a float and we'll call that the energy.

A

A

To give the path which is energy, hourly CSV, you already have it within the solution here.

A

And it has a header and a separator is all right. Next, we'll define our pipeline.

A

Context I transforms- and this is going to be, detect spike by SSA, which is singer spectrum analysis and for output column with your name of and master mode, creates our prediction class here, energy prediction.

A

And this is going to be a vector type of two items within the vector and it'll, be a double an array of doubles, and we just call prediction.

A

So for our output column, with the name of with the name of energy prediction- prediction property- it's just gonna get the name of this, so we don't have to hard code it every time.

A

And in our typical we run the same name of it's going to be energy data, that energy.

A

The next we have a series of kind of hyper parameters and I'm. Just gonna kind of fill these in you're free feel free to kind of play around with these parameters and oftentimes playing around with them can give you a better kind of a model or transform in this case. So the first one is the confidence.

A

I'm gonna say take things that are 90%, confident you're, saying confidence. Do the training window size.

A

I'm just doing 90 in the seasonality window, size I'll just do 30. The last one is to p-value history length.

A

Just leave that a 30 as well.

A

There's a pipeline so now we can generate our transformed data, a collie pipeline that fits on our data and from there just to transform on that same data the next we can create a predictions from our pipe on so they're contacts data that create numerable. It's gonna be give the transform data. Let me tell it to reuse.

A

The row object is true and then I'm gonna set it to a list.

A

We are fitting on our data for this pipeline and that data that comes in this is gonna, be an odd data view, and what this is doing is just basically transform the odd data view and to an innumerable if over it, alright. So.

A

So we want to get some columns for my data here.

A

It's the first one, get the energy column so get that for my for my regular data here and we have a get column method we can use and that's going to be a float in the name of energy. This is before we transform it by the way to an array and then the date column.

A

This is just so, we can see fiction if it's an anomaly a lot or not. If it is, we can tell what the energy level is and what date it is that came from it.

A

And so I'm gonna do a for loop next, so I can go through all the predictions.

A

A

Through each of the count predictions and at the prediction item within the loop in the prediction.

A

First item in the prediction equals one, and this is tells it if it's an anomaly or not, so this position will be 1 or 0 if this one. That means this an anomaly if it's zero, that means it's not an anomaly. It's just a regular value.

A

So we're going to write out the date and the energy item gets on the particularly row and then we're gonna just write down an anomaly. A lot I shouldn't always be 1 all right and then we'll do a console leave on so it stays up. I should do it and see what this looks like.

A

Okay, so we're getting some anomalies here, it's like quite a few, and so now we can actually go through this data and kind of look at each of the days here and so for 2:25 2018, we see at the hours of 6 7 8 & 9 have anomalies, so what can actually do is maybe cost referenced and of what was going on a day and see if anything may have caused that anomaly.

A

Alright, so that's it for this one, just a kind of a quick video to show how to do some anomaly: detection with an ml net and until.