DFFML Examples, 9 Oct 2021

Previous Meeting

⏯

youtube image

►

From YouTube: Use Case: Tuning Models

Description

Setup: https://www.youtube.com/watch?v=y6bz-9U4hJg
Make sure you're looking at the version of the notebook corresponding to the version of dffml you have! (dffml version)
The following is the link to the notebook from this video: https://github.com/intel/dffml/blob/master/examples/notebooks/tuning_models.ipynb

A

Hi there welcome to dfml videos in this video. We are going to go through the use case. Example for tuning models, we'll be looking at how to tune models by mutating configs manually and using tuners.

A

If you want to go through a detailed description about importing packages, downloading datasets and loading them into sources plugins, you may go ahead and watch the video on it, which will be also linked in the description for this video. We are going to head straight into creating our models.

A

The first step here is to instantiate the model you can use the model.load function to load any model in the dffml api. By passing its entry point to find out about all the entry points, you can go ahead and check out the model plugins in documentation.

A

Here you can see that all the entry points are given for every model.

A

So once the model is loaded, we instantiate it with the appropriate parameters. This includes a list of all the features in the data set, the target feature and the location or directory to save the model in you may also add any hyper parameters here for model specific parameters. You can check out the documentation of that specific model.

A

Once the model has been instantiated, it's ready to train now we will use the high level api to do this note that we also use the unpacking parameter unpacking operator here to pass each record in our data set as a separate argument.

A

Once the model is trained. uh We test our model using the accuracy method from the high level api to assess the accuracy we need to load and instantiate a scorer as well.

A

Let's use the psychic classification accuracy score uh by the entry point ac score. You can find more scorers and their entry points on the documentation as well as you can see. Different scores are provided on this page and their entry points are mentioned as well.

A

uh For this example, we will be using the ac score score once we have the square entry point and we instantiate the score. We pass the model, the score, the target feature and our unpacked test data set to the accuracy function and we get an accuracy of 0.75.

A

Since we didn't pass any hyper parameters at the time of instantiation of the model, it was trained on default values of the model. Let's try to tune it with a good, old-fashioned, hidden, trial method. We can mutate the hyper parameter values by changing them in the config. However, you might want to refer to the model's documentation first to see which of the hyper parameters are mutable.

A

So here we use the context manager to allow the mutation of the config and we set values for parameters of interest, let's once again train and uh train our model and calculate the accuracy, and would you look at that? We successfully decrease the accuracy of our model.

A

Well, uh we can try a couple other values uh in the spirit of trial method,.

A

So, let's just change a few values and random, and let's do it again now we see that uh it, it increased a little, but it's still lower than the accuracy on default parameters. Well, uh this could take all day. Fortunately, uh for us, tffml has the perfect solution.

A

We can use tuners to tune the model for us, uh for the sake of this example, uh we'll be using uh the parameter grid, tuner parameter grid takes a grid of different uh hybrid parameters and as many values as you want to tune from you passed in all the relevant data, such as the model. The target feature, the score: the train data and the test data.

A

And it tunes the model against the hyper parameter grid. Sets the model hyper parameter. uh Train set the model to the hyper parameters that are the tuned hyper parameters and trains it and gives you the tuned accuracy. As you can see, all the permutations of the parameter grid that we sent into the function, it's checking out all the accuracies and will provide you with the best version of the model.

A

Let's just wait it out.

A

I've actually enabled this info std out so that we can see what's going on inside the function.

A

We're getting the logs of every permutation okay, so here we have the results. uh The optical, optimal hyper parameters are learning learning rate, 0.4 and estimators 150 and max depth for accuracy. Optimized of the optimized model is 0.60, so there we have it and our model is set as well.

A

Now uh here is a small visualization of our model that our unjoint model and the tune model, and we can clearly see that the accuracy increased here. I hope everything was clear up till now and that this tutorial was helpful, uh feel free to open issues on our github. If you have any queries or you can also reach us out on the getter community. Thank you.