.Net Foundation ML.NET, 3 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Find and Replace Missing Values in ML.NET

Description

Learn how to find and replace missing values in your data in ML.NET.

Code - https://github.com/jwood803/MLNetExamples/blob/master/MLNetExamples/NullValues/Program.cs

ML.NET Transforms Playlist - https://www.youtube.com/playlist?list=PLl_upHIj19ZyJf-ItL5Mey9mniPiTKtvv

ML.NET Playlist - https://www.youtube.com/watch?v=8gVhJKszzzI&list=PLl_upHIj19Zy3o09oICOutbNfXj332czx

Contact:
Twitter: https://twitter.com/JWood/
Blog: https://jonwood.co/

Gear used (affiliate links):
Mic - https://amzn.to/2YEXtxI
Mouse - https://amzn.to/2ZtASoQ

A

Oftentimes when doing machine learning, you're gonna have data that gonna have some missing data within it and it can to be cleaned up before you can put it in a machine learning algorithm. Otherwise, it's gonna error back at you.

A

So in this video I'm gonna show how you can handle missing values with an ml net, specifically I'm, going to show how you can indicate missing values, and then, once you indicate those how you can replace those missing values, all right, so I have Visual Studio loaded here with the console project and already have em on the net installed version 1.4 and we're doing the housing data. So in order to have the input schema loaded as well, so we can start loading that data in.

A

Like always, the first thing we do is to create a new email context.

A

And then we're going to read in our data from the CSV load from text file using the housing data schema on just showed earlier.

A

As a header and the separator, it's going to be a comma next I'm going to give my columns here so I'll use the data. That's schema. It's going to get the schema that it loaded from the file. Then I'm gonna use some link magic to select the column, names and I'm going to filter out.

A

With a call name, there's an equal label and what I call name? That's an equal number column name, doesn't equal the ocean proximity.

A

So we don't want it to include our label column or the ocean proximity which is a categorical column and now just set that to an array.

A

Next, to indicate our missing values, a.

A

Coral pipe line here and I'll use a transform to concatenate all the those columns into a features, call them and next use. Another transform go indicate missing values.

A

And similar to some other transforms that we used it has an output call name which I call missing values and the input call name which scummy features.

A

And with that pipeline we need to fit the data with it. Then I'll go ahead and transform the data without python and that produces a and add that data view, and so we can look at what that data is I'm, going to create an enumerable from that and I'm going to put it into a new class here and called missing data. I. Don't define that in a bit but I'll pass in here.

A

The null values not tell it to reuse the object as false and I'll set that to an array as well. So let's create this missing data class here.

A

What I'm gonna do with this is I'm gonna inherit from the housing data. So it's going to append on to the housing data items and with that I'm just kind of a single property. It's gonna be a boolean array. You know it's gonna, be called missing values and now I can traverse through this data here, since it's an array, I'm gonna do a for loop and it's going to be the null data that length and if.

A

That no data, the missing values on that has any where it's true.

A

And get the indexes of all the missing values.

A

Because it comes in it's going to come in for each column, that's going to come in whether it's true or false, so I'm checking, if any of them. If any columns come in true right here, I'm going to give the indexes of the columns where it does equal. True spacing values, I'll select, get the value and the index, and so if there is a value I'm going to give the index otherwise I'll do negative one and I'll filter those where the index does an equal negative one.

A

So I want to get the indexes that have missing values and I'll set that to an array and for each of these items each of these indexes. They have indicated that are missing and get the feature by looking at the columns based on that index and I can write out.

A

What feature was marked as missing and I can tell it what row it's in.

A

Since we have this index up here,.

A

Here we go now, let's do a console of real on, so the console doesn't disappear on me. Let's try running this and see what we get here.

A

And, of course, once again forgets to mark the CSV file after copy, so try that again.

A

There you get so housing median age and the first row has a Miskin value. The total rooms and row four has a missing value, so we can double check this, so housing median age, no value in a total rooms has a no value sweeps. Another we indicated are no values. Let's replace these items.

A

For that I'm going to create a new transform for it, so I'm going to do the same. Concatenate, transform and I do another transform called replace missing values.

A

My upper column be new values, you're, not passing the features and then there's another parameter, which is going to be the replacement mode. That says how you want to replace these items. I mean your default value, which is going to be the default of whatever type the column is. So it's a flow. There's gonna be zero point zero and if it's bullion, it's gonna, be false, and so on you do the maximum value. Let's end, I call them the mean value of the column or the minimum of a column.

A

I'll just do the mean value here then, with the transform.

A

We will call fit on it.

A

With our data and then transform on it and then like a above, we're gonna call the create enumerable. So we can look at it, and this calls for another class here kind of like we did up here for the missing values, I'm going to create a needle and called replaced values.

A

That takes in replaced values and we use object as false and I'll call to array on it. Let's create this replaced, values, class and we'll do the same thing. We're going to inherit from the housing data. Do a single property. It's going to be an array of floats and called new values and I'll do a for loop and for each and other place, data that Ling and for each index and the missing values items, but unfortunately, there's no way to get that right now.

A

So what I'm gonna do instead is I'm going to create a class field here private static, it's going to be read-only, it's gonna be a dictionary.

A

It's gonna be integer key and energy value. It's gonna call it missing indexes.

A

And then down when we get our missing indexes for each of that indexes, I'm gonna, add that row index and the missing value index into it so now down here for each of those that I get if the current index matches the index that key. So it's on the current row that has the missing value, we'll write out our.

A

New value replaced data for that index. The new values at.

A

The missing data value index here just run that and see what that looks like all right. So we have our two missing values that we indicated our first one is a new value. Is the median house age the mean value of lead and the next one is the median value at the total rooms, all right everyone. So that's kind of a quick way to on how you can indicate your missing values in your data and how you can replace those missing values. So I hope you enjoyed the video and we'll see you next time. Thanks.