.Net Foundation On .NET, 20 Aug 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Training Machine Learning models with ML.NET

Description

ML .NET Homepage: https://dot.net/ml?WT.mc_id=ondotnet-c9-cxa

ML .NET Tutorial: https://dotnet.microsoft.com/learn/ml-dotnet/get-started-tutorial/intro?WT.mc_id=ondotnet-c9-cxa

ML .NET samples on GitHub: https://github.com/dotnet/machinelearning-samples?WT.mc_id=ondotnet-c9-cxa

ML .NET Model Builder extension for Visual Studio: https://dotnet.microsoft.com/apps/machinelearning-ai/ml-dotnet/model-builder?WT.mc_id=ondotnet-c9-cxa

ML.NET allows .NET developers to easily build and also consume machine learning models in their NET applications.

In this episode, Bri Achtman joins Rich to show off some really interesting scenarios that ML.NET and its family of tools enables. They talk about training models, AutoML, the ML.NET CLI, and even a Visual Studio Extension for training models!

1:40 - What is ML .NET?
5:19 - How can I load my data into ML .NET?
6:55 - Sentiment analysis model creation demo
10:54 - Model training and consumption
15:30 - Rich's ML validation test
16:37 - Object detection demo
18:53 - How are customers using ML .NET?
22:21 - Using AutoML and the Model Builder extension for Visual Studio
25:06 - Using AutoML with the ML .NET CLI

Get your questions answered on the Microsoft Q&A for .NET - http://aka.ms/dotnet-qa
Learn .NET with free self-guided learning from Microsoft Learn: http://aka.ms/learndotnet

A

Hey, we're here with bree, and we're going to learn about machine learning with .net.

A

Hey, welcome to another episode: on.net.

A

today, we're going to be talking about ml.net with bree who's, one of the people working on that..

A

So how about you introduce yourself,, then we'll get into the topic? sure.. I'm bree,.

B

I work on the.net, team. currently,, focusing all my time on ml.net for about the past six months.

B

Ml.Net was actually released at last year's build and this year,.

B

We finally went into 1.0. yeah., it's always fun. Working on that.

A

1.0 release because there's usually all these foundational decisions,.

A

Some of which will never get undone,.

A

That's the one we're going to go with for the rest of time.

A

So I imagine, there was some of that that went on leading up to 1.0?

A

definitely, yeah., a lot of stabilization in the api.

B

We also added a really cool new tools that are in preview, that I'll talk about a little bit later.. Can you talk a tiny bit about,.

A

I know there were a bunch of different teams that were involved,.

A

Was there a bunch of give-and-take.

A

Or was everyone moving in the same direction?.

A

Or a little bit of both?.

B

A little bit of both definitely., I think different teams have their different opinions and especially with all the different offerings, we have at microsoft, for ai and ml,, it can can get a little bit difficult sometimes,, but in the end, it really came together into a really great product and worked really well together.

B

awesome.. So what do you want to talk about? First?.

B

I guess I should explain a little bit about what ml.net is.

A

B

Basically,, it's just a free open-source, cross-platform machine learning framework, for.net developers.. Our goal is really to make.net great for machine learning and a lot of different types of scenarios.

B

I was going to ask a question: first, right. go ahead. which is,. So there are these other machine learning things.

A

So I know: there's tensorflow,, there's a bunch of stuff in python, space,.

A

You might say some of their names., then there's this onyx thing.

A

Then there's probably other machine learning ai things that even microsoft makes available..

A

So before we get into the details of ml.net,.

A

Where would you say it sits?? Is it competitive with all of those things or complimentary?.

A

I wouldn't say it's competitive.

B

What makes it unique is that it brings machine learning into the.net ecosystem., so existing.net developers can then use.

B

Their existing c# and f# skills to integrate machine learning into their.net applications., so they don't have to go and learn a new language. In order to do this thing with machine learning, which before, they didn't have to. a lot of times, when they had to do that and move to other languages are tech stacks,.

B

It would be difficult to then integrate that back into their.net applications.

A

One more question related to that hesitant though,.

A

We're going to go into the winds,, but there's also this tensorflowsharp thing.

A

one can describe that as bringing machine learning into the.net ecosystem,.

A

But I'm somehow guessing that tensorflowsharp and ml.net are not exactly the same.

A

They're not. well,, especially because ml.net.

B

Is being worked on by microsoft., so tensorflow is actually for deep-learning,, whereas ml.net right now supports classical machine learning scenarios.. That would be things like sentiment. Analysis, price, prediction,.

B

And deep learning gets a little bit more into image. Classification, object detection..

B

So what tensorflow on onyx do is they were able to extend them.

B

All of them and use those in order to add those scenarios.

A

What I specifically meant was tensorflowssharp.

A

So there's this library,, that is a c# binding against tensorflow,, but your answer is probably still fine for that.

A

sounds good.. How about we look at your table?.

A

Sure. so I've already.

B

Talked about how this is built, for.net developers., so you stay in the.net ecosystem,, but not only that..

B

You don't really need existing knowledge of machine learning, and I'll get into that a little bit later,, but we have these really cool tools that abstract away the data science from it.

B

so I'll get into that later..

B

It also just makes it really easy to create custom machine learning, models.

B

so right, now, if for instance, with cognitive services or things like that, they're, pre-built models that you use on your data or you used to make predictions..

B

So what ml.net you can do is make your own custom model with your own data.

B

So it makes it more specific for your scenario.

A

B

I already talked a little bit about how it's extendable with tensorflow and onyx,, and things like that from even more scenarios..

B

What's really cool is that ml.net is not actually just a year. Old.

B

It was actually used in the company, started by microsoft, research for the past eight years..

B

It's used internally, a lot of huge products, power, bi,, key influencers,.

B

Outlook, meeting insights being suggested search,.

B

And the list goes on., so it's been used here for a really long time.

B

what ml.net did or what we're doing with ml.net,.

B

Is making the api friendlier and open sourcing it so that other people can then use it. makes sense.

A

So one high-level question based on what you just said,.

A

Which is you said, there's some pre-built models that you can just use as is,, but you can also train..

A

You can build new models based on your own data.

A

well,. Imagine I've got,, I'm team terabyte.

A

I'm just making up that number sql database.

A

It's got all this data in it.. How do I think of connecting all these rows and.

A

Tables in sql server to ml.net?.

A

It's actually you can., you can.

B

Load in your data from a file or from sql server., what you do is you load it as an innumerable., okay.

A

So you can do it from streaming sources like that.,.

B

I see. that works pretty well.

B

A

I don't know if you have any numbers probably,.

A

Something smaller than I'm team terabyte..

A

Do you have any numbers that talk about how long it takes to.

A

Train a model with say a certain size of sql server?.

B

I don't think of the numbers off the top of my head..

B

I know we did train,, I think a terabyte of data and it took a few days. okay. that was with model builder,, which again, will be coming. Up.

A

Okay., so that gives people a sense.

A

now, if your training,.

A

This is that activity that one to two day thing for that large that terabyte, that's pretty big..

A

That's not something you're running in like a cicd flow.

A

That's something that's more of a one-off activity that you do.

A

Then you get this model out of it, and then you check that in somewhere, and then you run with that for a while until you decide to replace it.

B

Yeah, exactly. okay. I'll, actually show you a little bit of code. Here.

B

First. so sentiment analysis is a really commonly used. Example to show machine learning.

B

So I'll show you this. actually, this blazer app here,, which has real-time sentiment. Analysis., if you say something like "ml.net, is awesome.".

B

You can see the slider goes. Up., if you say something like "that is rude,". You can see the slider goes down..

B

So what this is doing is using an ml.net model in this blazer application.

B

We'll go into the code here.

B

Lots of it. lots of times [inaudible] .

A

After we get through this,, I want to try one.

B

Sure. lets go through this first. sure., so, what's really cool is the steps are the same for every time you train a model.

B

The first thing you want to do is create this ml.net environment or mlcontexts.

B

sure.. It's like dbcontext and entity framework conceptually.

B

Then what you do is you load in your data and.

B

This point: it's just this yelp_labelled.txt,, which I'll give you a little preview that.. You have your text here and then your sentiment here..

B

So one is positive and then zero is negative..

A

Right., so this is our data source.. This is like the sql database in this sense.

B

In this case, it's just a text file but yeah,. This is our dataset right here for training..

B

So then, what you do is you load in from a text file this,.

B

Where you have your datapath here, you're, specifying that there's no header.! If we look into the sentimentdata-.

A

This reminds me of mail merge. yeah.. I've actually used that very recently.

B

Okay. sentimentdata,, you can see here.

B

That all it does is it's a schema for your free datasets.

B

So you have your sentimenttext, which is a string, and then your label, or your sentiment that you want to predict is a boolean.

B

So that maps out here and you're loading in that data.

B

We're right here., then dataview is.

B

A way that data is represented in ml.net.

B

It's really flexible and efficient way for looking at tabular data, that's rows and columns..

B

So what we do is load it into that data view..

B

Then. What we do is we add data transformations.

B

the way that it is now with the text, that actually can't be accepted by machine learning algorithms,.

B

It has to be featurized into numeric vectors, which will be accepted by the machine learning algorithm.. So we've added this featurized text data transformation.

B

Here and we've added that to what we call our pipeline., then what we do is we choose our algorithm and this case you can.

B

See that we have quite a bit to choose from for binary classification,, which is our task for sentiment, analysis.

B

right now, we'll just choose sdca logistic regression.

B

I'm guessing gethashcode isn't one of them.

A

Yeah. then step four,. You just train your model.

B

So, as of now before you call this fit method,.

B

It has a lazy approach where you're just adding things to the pipeline.. Once you call this fit method on your data,, it actually starts the model training.

A

Right. one interesting thing is often we talk.

A

About a sync, with a lot of the dominant product,.

A

But these look like these apis are all entirely synchronous.

A

I think they were intended to be run in this like stand alone, batch process of the model.

B

So that's why they're synchronous. yeah. yeah., then this is an optional step that usually you probably want to do,.

B

You want to evaluate your model., so what I've done is taken a separate dataset, which is,.

B

It looks the same, but it's just reviews from amazon instead of yelp, and use that to get evaluation, metrics.

B

So you load in from the text file, you make predictions on that test data and then you get a variety of metrics here,.

B

And in this case, it prints out the accuracy..

B

So then what you could do is save the model at the end and then use that in any other of your applications., so that can be web app..

B

What else? you can do console apps,.

A

Desktop apps [inaudible] microservices and containers.

A

yeah. mobile app., yeah. right., so another takeaway is,.

A

I take it. The training app and the.

A

App in which you're consuming the model are always going to be different., not always..

B

Okay. they can be the same, especially like, for instance, you can make.

B

A single prediction in the console app that you are training to model, but in most cases they will be similar.

A

Okay., that would be not the most common case.

B

Right. exactly. so most of the time they're separate, apps., yeah, exactly.

A

This is either going to be a console app or just like a very minimal ui app that has probably relatively few buttons.

A

Right. yeah, exactly.

B

So what we've seen so far, and with all of our samples that we have on github, all the training is done in console apps.

A

Yeah. actually, back to my question about cicd,.

A

I guess I can imagine you and I set up this company together.

B

We're like super pro at this,.

A

And we want to have the best trained sentiment, analysis.

A

I could imagine that every night we have this batch job, which there's more data that came in through that text. Box..

A

We just rerun all of our models and then we just see if they have better results.

A

If it has a significantly better result, then sync on or maybe we should just deploy this new model.

B

A

Meaning it doesn't have to be the case that you only run this like once a quarter.

B

Right. yeah,, you don't have to do it that way. some people will do that., but if you're getting better data,, it's definitely better to add that into your dataset for training.

A

Right. I mean, even if it takes two days to run like you, said,.

A

I guess maybe we could just do it once a week.

A

yeah. then get a better model.

B

Yeah. okay., yeah. definitely., so I'll actually show you what this looks like when I run it., you can see right now. This ml models, folder's empty.

B

So we'll go ahead and start running that.

B

Maybe. there we go.

B

So you can see we added a few console lines. There.

B

We actually printed out the accuracy which is about 75 percent,, and then we saved the model.

A

Okay. from some exposure to the sun, the past,.

A

75 percent is probably good for the amount of time.

A

We took on that, but 75 percent in the general case is bad.

A

Or is that the wrong way to think about it?.

B

It's actually not.. It definitely depends on your scenario.

B

It helps to pair the accuracy with also trying out predictions.

B

In this case, if you try out a lot of predictions,, I know you wanted to try another sentiment one, and it's a pretty good dataset..

B

So it'll do pretty well. once you start getting into the negations like it is not good,.

B

That's where it has a little bit of issues.

A

I see. yeah., that's common for sentiment, analysis.

B

Okay., but you can see the model here is just a serialized that file..

B

If we actually come up back to this program.cs, where.

B

We actually consume the model or used the model and make a prediction, we've. Ml.Net is awesome, and that is very rude..

B

So, I've already made a reference to the generated libraries here,.

B

The class libraries here in our predict sentiment project.

B

I'm actually going to drag this up. Here.

B

So that it can use the model.

A

Right. that we just built.

B

That we just trained., then we'll go ahead and start that.

B

Then you can see, ml.net is awesome., it predicted it as a positive sentiment.

A

B

That is very rude, which is a negative sentiment.

A

Right., so I'd that you could build like x unit test for example,.

A

That did something very similar.

B

Yeah. definitely., we definitely can.

A

Is that what people do to validate.

A

The model or I guess, since the training,.

A

Yeah,, I guess how do people test?

A

so there are those evaluation metrics that I mentioned.

B

If we go back to,, I believe it was here where we trained our model and we come to metrics.accuracy.

B

You can see that we have quite a bit of different metrics,.

B

Which, of course, some of them might not make sense,.

B

But we do have in our docs, explanations of what these things mean and they are common data, science.

B

right. they're, all useful for different, reasons.

A

So that's one thing, but that is also the way you explain,.

B

It's also a way that people can do it.. I haven't quite asked around yet to see how people are really using it.

A

That's our next step is to.

B

See the different ways that people are using in the different scenarios,, how they're testing it in all the different cases.

B

okay. that makes sense.

A

So we're going to try my-.

B

Yes., let's try.

A

So it is, you're, being obtuse. okay.

A

I thought that might break it, because I think "obtuse" is not.

B

It's probably not in the data set that we used, right?.

A

Yeah., well, and "obtuse" is kind of an obtuse word.

B

Yeah. any other ones?

B

that was me being clever. yeah,, any other ones. You want to try? no,. That was the only one I wanted to try.. So that's just one of the many scenarios.

B

I'll actually show you here.

B

some of the other scenarios we have: product, recommendation, price, prediction, segmentation..

B

We have all these samples on our github..

B

If you clone that,, you can just try them out of the box.. I have another one. sure.. Let's try it.

A

He is acutely aware of his intelligence.

A

Wow., apparently, I'm good at breaking this thing.

B

Yeah., we may have to add that to the dataset now, input it on the dataset.

A

B

So another really cool example that I like showing is object. Detection..

B

This is one of my favorite ones.. You can see here.

A

Right,, so this is the bounded box. Scenario.

B

A

I assume, if you were doing some of these programs, where you have a photo collection,, will say, "oh,. These are all the pictures of bree in my photo collection.", I assume as the basis of that,.

A

They used something like this to figure out where the person is.

B

Yeah., I'm pretty sure.. What this is using is an onix model, actually., I believe, onix. Yellow three is what it's called.

B

This is actually in our- sorry,.

B

This is in our github repo,, so you can download this and try it yourself.. I changed out the pictures,, but you can try it on your own pictures..

B

You can see here that it's located this tv monitor, this bottle, and this chair from my focus. Room.

B

we've even got my living room here with a sofa and a potted plant.

A

Right., so the way this program is running is the idea that you click that button,.

A

And I guess those are already resident on the server.

A

But this is the pre-object detected version of them on the right.

A

Then you're getting those images served back.

A

To you, with the object detection put in them.

B

Right. exactly. it uses that trained model.

B

I think objects that it can identify are.

B

Like sheep and sofas and dogs and cats.

B

It's trained on specific things, to recognize, specific objects to recognize. for instance,. It might not recognize grass or a table or things like that., but I'll show you another one, just to show that you can choose any image. Here.

B

Here you can see it actually identified a boat from just the images that I had there. okay., it's the sort of thing where,.

A

At least with my understanding of this stuff,, if you trained it exclusively on white boats,,.

A

And then, all of a sudden it saw an image with a blue boat or a red boat,.

A

Then it might get confused.

B

It might, yeah. definitely,, it's better to have a variety.

B

Also, just giving it images of not boats, if that makes sense.

A

Having just a variety of both is the best for training, the model.

B

Yeah,, those are the demos I had., it's pretty crazy, the different scenarios that our customers have been using. This for.

A

Right., so you have customers?, we do. it's great., because you just released your 1.0 just now.

A

Is it the case that you just have this growing set of customers along the way?, or do they come mostly at the end or a little bit of both?.

B

We actually had some before we even hit 1.0,, which was really cool. I'll, actually get into a few of them, because it's some of my favorite stories here., my top favorite one- is evolution software..

B

I think they started with version 0.4,.

B

Something like that, and they've been upgrading ever since.

B

But essentially, what they do, is, they do. Commercial hazelnut, drying.

B

So this is a commercial hazelnut drying.

A

It doesn't sound like a software company. no,, it doesn't. this image.

B

Here holds 50 thousand pounds of hazelnuts.

B

the business problem they are having. Is, hazelnuts have to be at a certain moisture level in order to be profitable..

B

So if you over-dry them,, they shrink and you lose money.. If you under-dry them,, they can get moldy and you lose money..

B

The way that it works now, is, people have to climb into here as it's drying and do the sampling.

B

They take a bucket, and they take out hazelnuts..

B

They manually test to see the moisture level..

B

They have to do this every so often, and the conditions are less than ideal.. It's 120 degrees, fahrenheit, 100 mile per hour, winds., it's just not great..

B

So what the people at evolution software wanted to do was eliminate this manual process.

B

They used sensors to gauge temperature and pressure.

B

Then they used the sampling data that they had before as training data.

A

So what they can do is predict.

B

The moisture level of the hazelnuts based on all of that., they actually created this application. Here.

B

They use [inaudible] for real time, updates.

B

They use their own asp.net core.

B

because they have a lot of.net in their product.

A

They're totally all in.

B

Yeah, they are., so they actually created this for the operators that says, "hey,, this batch is ready., you should go, get it," or "this is already too much," or "this is how much you have left.".

A

It's in the danger, zone. right. exactly, in the danger zone.

B

I did not know that it has to be between eight and a half and 11 percent.. It's the ideal, moisture level for hazelnuts.

A

Yeah., it sounds a little bit like a humidor that.

A

People use for cigars that you sometimes see in stores.

B

Yeah,, it's similar.. Maybe someone will use ml.net for that. One day.

A

I'm actually not a cigar, smoker. yeah. then another case that we have,.

B

Brenmore is really cool.

B

They do surveys for patients after they come to do. Doctors, visits.

B

They collect that data and try to improve the patient experience.

B

What they had was all these surveys and all the survey data,.

B

And they have these free-form comments.. They found that it took a really long time to parse through them manually and then direct it to the correct personnel..

B

So what they're using ml.net for is classification of those comments. both to say if it's toxic or non-toxic,?

B

So is this a good or bad feedback,, but also to put it into categories such as experience,.

B

Facility, provider, staff, and things like that., then it'll automatically route to the correct personnel.

A

Sounds a little bit like github issue, routing.

B

Right. exactly., that's actually the one that we use..

B

We have that sample in our repos,, using multi-class classification for github issues.

B

One of their quotes here, actually, was-- they mentioned automl,, which I haven't talked about yet,, and I want to get into that a little bit., but they used one of our new tools called model builder that uses automl or automated machine learning..

B

What that is is it, essentially,.

B

It generates a model for you, based on your data and your task.

B

I'll show you with the sentiment analysis, one right, here.

B

The way that it works, is, you download the visual studio extension, and it calls model builder.. You open, visual studio.. All I've created is.net core console application.. There is nothing else., just "hello, world.". What you do is you right-click,.

B

And you say: "add machine learning.".

B

You have a few different scenarios right now..

B

It supports classification and regression scenarios.

B

So we'll go ahead and do sentiment analysis, because that's what we've been doing., you can do it from file or sql server..

B

I've got a file ready,, so we'll go ahead and do that.

B

Wrong file. here we go.

B

What you can see here is a little data preview where it has the sentiment and then the sentiment texts, just like we saw before., it's just a different dataset.. What we want to predict is the sentiment..

B

So once we get this model,, we want to be able to feed it sentiment, text or comment. Texts,.

B

And then get the sentiment back.! That's what we're going to predict here. we're going to move on to the "train" step, and we're going to specify a time to train.

B

we're going to leave it at 10,, which is the default.. If you have more data or you want to use it in production,, usually you want it to train for longer..

B

What it's doing right now is it's using automl to iterate through different algorithms, data, transformations, and algorithm options to give you the best model or the highest-performing model..

B

You can see here., it's all the models that it's going through,, the one that it's found as the best so far, and actually the accuracy of that model..

B

If you go to the "evaluate" step,, it shows you a few evaluation. Metrics. accuracy is a pretty good one to gauge..

B

You can see that it shows this averaged perceptron binary,, which is actually different than the one that I had before., but I don't know much about algorithms,, so it takes care of that for me.

A

It all sounds good. yeah., what's also really cool is,.

B

Once you go to the "code" step,, it will actually generate the consumption and training code for you..

B

So you add those projects, and you can see here. It adds these to the right.

B

You open this model builder, and you can see the steps are very similar to what I had manually written before,, but it just generates..

B

I don't know how these are happening. Here.

B

Then this one, you can actually run your model with program.cs.

B

Lots of red, perfect.

B

it generates those class libraries for the sentiment and sentiment text,.

A

And the predicted label here.

B

Nice. so then, what you can do is go back in here and use your model.

B

If you go back to model builder,, it actually has that code that you can just copy-paste, over, and you're, able to use your model. right.. So this is,.

A

Obviously,, a visual studio plugin that gives you this experience.

A

Are you thinking about any outside of visual studio experiences for the model building?.

B

Yeah., we actually have that ml.net cli,, which does the same thing.

A

Actually, ml.net cli is behind the scenes here in automl.

A

This is just a ui put right on top of that. I'll, actually go back.

B

Because, obviously,, if we're back to our example,.

A

If we're wanting to run this once a week on an ongoing basis,.

A

We probably wouldn't want to do it manually in visual studio.

B

Right. yeah., we have that option, and it's really good,.

B

Especially if we're getting started,, but we also have it there on the command lines., so both ways. awesome. in that way,, it's cross-platform as well,, so mac, linux, windows..

B

You can use automated machine learning.

B

awesome. okay.. Do you have.

A

Any closing thoughts or things you'd like to share?.

A

Where should you go if you want to get started?.

A

Yeah. to get started,, it's very, very easy.

B

I'll type it in here for you, but dot.net/ml.

B

That will redirect you to the pages I just showed you, maybe.

A

I think maybe those red squiggles were maybe due to some internet connection, problem.

B

Yes,, I'm sure. I'll just bring it back. Here., there's a big "get, started" button.. This will lead you through how to install model builder and get started with it, and also the cli..

B

So, whichever way you want to get started, there.

B

Then another great way is to just go to our samples on github and just try those out.

B

a lot of people,. The way they started was downloading one of the samples and then just adapting to their own scenarios..

B

Any of those ways,, it's super easy to get started.

B

If you have any feedback, please, on the github,, let us know.

B

yeah, a file,, an issue or whatever.

A

Okay. awesome. well,, thanks for being on the show and teaching us about ml.net.

A

sure. thanks for having me. okay. well,. This has been another episode of on.net, and I hope you learned something about machine learning., thanks.,.