Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hyperparameter Tuning Using Kubeflow - Richard Liu, Google & Johnu George, Cisco Systems

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Hyperparameter Tuning Using Kubeflow - Richard Liu, Google & Johnu George, Cisco Systems

In machine learning, hyperparameter tuning refers to the process of finding the optimal constraints for training models. Choosing optimal hyperparameters can drastically improve the performance of a model, but the search space grows exponentially with the addition of new hyperparameters. A closely related subfield of automated machine learning is neural architecture search (NAS). In recent research, networks generated by NAS algorithms can even outperform handcrafted neural networks. However, like hyperparameter tuning, the process can be time-consuming and expensive. We present Katib - a Kubernetes-native automated machine learning platform for hyperparameter tuning and NAS. Part of the Kubeflow platform, Katib offers a rich set of management APIs in the form of custom resources. We will demonstrate how to configure and run an experiment and compare performance in Katib’s UI dashboard.

https://sched.co/NrnY

A

Aha, you know, you sure, do good G, Valenti or so nice little vertical. My shirt was yours for a slip it on gonna, assume the dong I'll get a door and y'all sure what they take. A ho Speaker John retort is from Cisco wanna.

B

I'm Journal, like technically it from Cisco I'm walking in cue flow, related.

A

Okay, so I'll speak to English now, so today's agenda we're going to introduce a hyper parameter to knee and we'll be briefly covering what it is and why it's hard and we'll go into our main topic, which is a coupe flow and cakap and we'll see how these systems would help us to hyper forever tuning and will give a brief talk about the system.

A

Architecture and the workflows now will also show a quick demo of our system and then we'll go into a new architecture search which is a new area of research and implementation that Pat has done and finally we're going to some future work. So what is a hyper parameter knee? So, let's start with an example: okay suppose that we're doing this, this is a very basic of machine learning program for recognizing digits right. So, on the right hand, side you can see some training code and, in fact, I'll rely on the bottom there.

A

You can see that, though this is where we were starting to fit a model right. So hyper parameters are the parameters that you set before the training process begins. So if you look at this list, we have your network batch size, number of epochs, the learning rate and so on. Okay, so these are the parameters that are that governing the training process, they're external, to the training model right. This is, in contrast to model parameters which is learned by the your training.

A

So yeah so high parameters, configurations variables, are external to the model and they're set before the training begins and so setting the right parameters can really significantly improve your auto performance, but it's only if you do it correctly, which can be very difficult, so hyper parameter tuning is a process of finding optimal values for other parameters such that your objective function. In a previous case, that is our function to predict gages a handwritten digits. You want to maximize the prediction: accuracy, okay, so why is it hard?

A

First of all, if you increase your number of hyper parameters and your have a parameter ranges, the search space becomes exponentially large right, so in previous slides, we've seen about six or seven parameters. So imagine if you have a more complicated model and your your your space is just going to grow, so turning by hand is really inefficient and it's very prone to errors, and you want to in order to really track your performance from one computer to another.

A

You want to track metrics across multiple jobs, so that means you had to have some kind of a data storage, some kind of a visualization interface right to see what your experimentation has done. Maybe you want to anchor on some particular value for one parameter like the patch size, and you see what changes when you adjust the learning rate right, so you need to be able to track matches and also you need to be able to manage resources, infrastructure, so training jobs. Sometimes they acquire additional.

A

Maybe it may be additional hardware, some completing resources, so you want to be able to allocate and to provision and to clean up the resources and finally, there's a lot of different frameworks and algorithms to support, for example, just from training the example that we've seen is for MX net, but you can have a tensor flow or PI torch model and algorithms that could be a grid search or a random search for Bayesian optimization as long right. So there are lots of different variety of similar problems.

A

So how does a kubernetes help us so first of all, criminales makes it really easy for you to build micro services, so micro services are self-contained and lightweight services that do one thing really well and kubernetes also makes containerization easy with docker right. So the good thing is that really applies to our model of a parallel training jobs right, yes, uh jobs. We can containerize them, and this makes them very easily scalable, and this also helps our system build our resilience right because a coburn editor raisa, auto healing and recovery options.

A

So it's a meaning that the failures of a particular training container- it's it- won't affect the whole system and the system can actually automatically bring the back up. Okay.

A

This also helps us with our a problem of multiple frameworks and different algorithms to support, because we could create this generic interface. That's interacting with the outer atoms as microservices, so we'll go into more details about that in a bit and coronary is also pretty good at describing the desire state right. So we call those a declarative, API s-- versus a imperative right. So this is the remember we talked about.

A

We want to manage resources right, so this makes managing resource very easy because we, just all you need to do- is describe the desire state instead of having to write a complicated instructions on what you want to provision.

A

Kubernetes has a very flexible API, meaning that we can extend the core API would cause some resource definitions, so this allows us to interact with the objects using this rest, api sand tools like poop, taro and finally coronaries lives is very portable. So in production environments we often have to deal with different environments, like you could doing.

A

Local development versus doing on-premise, hosting or even molten cloud I prefer Nettie's has various options, but if you build on top cover an area, so your application should run everywhere with your kubernetes cluster is deployed so a little bit about coop flow. The cupola is coronaries native machine learning. Problem for developing orchestrating deploying adjoining scalable end-to-end machine learning were close. So that's that's quite of a quite a mouthful right.

A

So if you look at this diagram and there's not a skill but a lot of problems with building machine learning in production, do it deals with the things in the blue boxes right, so monitoring, service infrastructure and analysis, processing management tools, and so on right, your M, your machine learning core logic might be just only a small part of it so who Flo wants to make building all the other stuff easier. So you can concentrate on the core machine learning logic and so right.

C

A

Turn over to John, who will go to us, introduce us to cat it.

B

Hi everyone, so let's talk about cat it. So cattle is the hyper parameter tuning component in queue flow. It is completely kubernetes native. So that means you could install it even without cube flow. But then, if you want to use features of your flow like the distributor training operators, then it is advised to use within the cube flow ecosystem. So it is inspired from the Google Vizier project. You would have read the paper. It is a black box. Do you know you could that link is given in the bear and the below command?

B

You could trade that it's fully open source and it is framework agnostic, so which means you could use your programs based on your choice. It could be written internship by thoughts, MX net. You could just provide the images and it should start running. It has a customizable algorithm back in by default. It provides a random search grid, search based in optimization high program user could plug in any algorithms based on their environment, and you don't need to even restart the components.

B

It could be a status, a separate micro service and it should yeah without even restarting the component. It should start running. So, let's talk about the concepts. The the first concept is experiment, so it's a end-to-end process for hyper parameter. Optimization so, for example, take example of a digits recognition model, so you want to find the best hyper parameters for that. So that's the entire, the experiment, the users provide experiment, conflict and submit together, so it has different sections.

B

The the important sections are, one is objective, which is what we are trying to optimize, for example, for your model, you want to optimize accuracy and what you want to achieve is 99%, so the objective metric name is accuracy and the value is 99%, the second, with the search space, which is the constraints for configurations, so you could set limits for your parameters, say the max maximum or the minimum or if it is categorically could set the list of options that are valid for a particular option and the last one is the search algorithm, which is how you find the optimal configurations.

B

So in the last slide, as we said, you could use the default ones or the ones which is added by the user so like any other resource in kubernetes. Experiment is also a resource to be specific, it is a custom resource, so you could use all the standard api's to basically get create and delete the experimental source. You could use q cuttle, like any other source in kubernetes and the state is stored in kubernetes database and the life cycle is managed by the kubernetes controller pattern.

B

The second one is sedition, so once user submits the experiment config through kubernetes api, the experiment, controller picks it up and contacts the right suggestion service. So suggestion is one proposed solution to optimization problem, so you have set the search space and the suggestion is provided turn the based on the settings given by the user, which is one proposed solution by this, which is given by the suggestion service.

B

So each suggestions algorithm is a standalone micro service, so which means you could add that during the runtime and it should not affect the already running experiment and it allows user to create customized solution. Algorithms and the third one is the trial, so ones suggestions are provided by the suggestion service. The trials are created by this experiment. Controller, so trial is one iteration of the optimization process, so which means once you get the parameter assignments or the addition.

B

There's a worker process which actually starts running it and emits out the observation matrix so similar to a trial similar to experiment. Trial is also a custom resource and the experiment controller is the one which is spawning and managing the trials and, depending on the trial kind, you can have the distributed job for it.

B

So this is a basic workflow of the hyper parameter tuning. So, as I said earlier, this is the one optimization loop and this loop continues till one of these conditions are met so either the objective value is reached, for example, or once your accuracy has reached 99% or if the budget has exhausted. So you could also set that my maximum number of trials to be run by this experiment is X, so once X is reached or when your objective is met, the optimization loop ends till that time.

B

The suggestions are created and for the each suggestion trial is, trials are run and the metrics are emitted and these metrics are collected and new suggestions are created. So this is a system architecture, so the user submits experiment and experiment controller is the kubernetes custom controller. It picks it up and talks to the right suggestion through a catted manager, so algorithm related settings would have been given in the experiment and once the experiment controller gets suggestions, it creates trials. The trial control up again, picks it up and creates, starts executing trials and starts emitting metrics.

B

The metric collector is a separate process which again during the lifetime of trial, it collects the metrics and writes to the data store, and this will be again used by the suggestions for the for the future suggestions and that loop continues.

B

Yeah, due to network, we are not showing it live, but again we have a cacti Bui, which actually provides much of the features in the as a user interface, but for much much more customizable options. It is better to use the command line at this point, but we are continuously refining it. So, as I said from the user point of view, what we have to do is setting the experiment, conflict and rest all taken. Rest all is taken care by the cata controller, so the things the important sections that has to be filled would be.

B

One is a name, the name space where the experiment has to be run. The next section is the common set of parameters. One is the parallel trial count, which basically is how many number of concurrent trials have to be run. So, based on your resources, you could set a higher number of trials to be run concurrently. The other one is the max trial count, which is a budget saying. This is a maximum number of trials which would be run. They are during the experiment, lifecycle.

B

The third one is the max failed trial count which says maximum number of trials that I can have during the entire lifecycle. The next is the objective which says what is my type, whether I have to maximize or minimize the objective. Metric name is the metric that I have to optimize, and the goal is the value that I have to reach. So this actually means I have to maximize validation. The accuracy to 99% and I can actually provide additional metric names if I want to collect extra metrics during the process.

B

The next section is the algorithm section where I can specify the list of algorithms. This some some are already provided. You could actually add a deal during a development process. I could add extra algorithm settings which would be taken by the algorithm, and the next section is the parameter section where, as I have told this is. This is a place where you specify the source P for each parameter.

B

I have to basically say the max and min value, or if it is categorical, I can basically say the list of options that are valid for it, and the last option is a trial spec, which is the most important one, which would be where, where you would be specifying. What is your the actual job that I have to run.

B

So this is like the one run that that we have run using the base in optimization. So you could see that the final results have come to 99% converged to that and for each the trial, which we can see that what are the parameters of parameter values that have given to that result, and you could basically say which one and you can even go backwards and figure out what are the best values that I have to use. It.

B

A

So a little bit more about the this experiment, so this was conducted using about a hundred different trials and this is using Bayesian optimization. So this is one of the most common algorithms for Hyper for tuning. The ideas like the previous results of the trials are used to optimize the future results. So, given that we have selected some parameter values that have led to a clear results, we want to improve upon that are our current status.

A

So if you asfar hard to see from this picture, but if you look at the corner there, we started with maybe about 96% accuracy and we're able to improve it to 98 99 %. So and if you click on each of these trials, you actually you can actually see how the trial progress over time. So the blue line. There is the accuracy that has improved and there's also the validation accuracy. So in this case we use the validation accuracy as the objective metric.

A

So we use it use that to determine how good our current set of parameters is, and there, as Donna mentioned before, there are also ways to collect other metrics. So we can also see how it correlates with for that, which is the accuracy okay, so.

A

Okay, so I want to bring us back to a broader landscape so- and this is a problem with classical versus automated machine learning.

A

So in classical machine learning, a human expert gets involved in a lot of steps like selecting features, choosing algorithms, configuring, hyper parameter, values, evaluating performance, tuning models right and a general trend that we've seen in computing is more and more tasks become automated and more and more become dedicated to machines, rather than having a human getting involved in every step. So- and this is why we have this research field of automated machine learning, which is having a program generating the model without human Internet to reduce all these time-consuming steps.

A

So what we this well have you seen is a hyper parameter optimization, but in the general landscape there are also other areas like feature engineering and architecture, search so and I've included a link to a list of all I know papers, and this is a available on github and it's very very interesting. So there's something in common with all of these research right.

A

So in all the cases we're dealing with automatically generating a compact version space, so that can include the features, hyper parameters, architecture and, in all these, we generate some kind of a metrics to judge their performance and based on that, our optimizer algorithm generates a new configuration, and the output of this is a model which we can use to serve predictions.

A

So, in the recent development we have expanded catted to handle newer architecture search. So this is in contrast to the hyper parameter, optimization problem that we just mentioned, and a differences in hyper parameter search which only generating values for a list of parameters. So you can think of this as a vector search and your architecture search is a graph search.

A

So the space is exponentially higher than that than that of is is the one dimension higher right and there are a lot of research in new architecture search, so there could be searching for an optimal network or could be searching for optimal cell from which to the rest of the graph is generative, and there are also other that there are also various evolution strategies such as a PI generation. Apart by modification.

A

So that workflow for architecture search is similar to high performer tuning, except that we still have our objective conditions like whether we have reached our objective foundation or whether we have exceeded our search budget, but we've also included the constructing the model, because in ns the human is no longer that the human expert is known on providing the model right. So what we do is that the suggestion would or generate a list of evolved operations and from that we'll generate the next version of the model and then similar to type remember tuning.

A

We will run these trials based on the model that we've constructed and use a metrics to using the reporting metrics. We can just charge their performance and come up with the next suggestion for the architecture. So currently we are supporting ml of net and reinforcement, learning for n NS.

A

Okay, so we have seen what khatib has been able to do for now, but what's coming in the future, so first of all, we want to provide better production, ization support in the architecture diagram. You may remember, there's a component for data storage. This was useful because sometimes you want to store your metrics for net the next run or maybe to resume with experiments or maybe just as backup, but currently cat AB requires an instance of my sequel go to run as the data source back-end.

A

So one thing we want to do is to support customizable database.

A

Another thing is metadata integration. This is in Baris cope, and this is a part of the broader Kukla team effort. So what we want to do is suppose that you you've you've done some hyper parameter to experiments and you have your able to produce some pretty good results right, so you want to take those models. Have you train and serve them right? So this requires some kind of a generic store where you can store your metadata and to be used by other parts of your machine learning pipeline.

A

Finally, we have supports for other features like long-running experiments, so suppose you have a tried to some combination of parameters, but you want to change your of them a little bit. Maybe change your budget and you want to see you want to continue experience from before. So these these. This will provide better support for for a high performing of service imp adoption.

A

Another aspect is that we want to add more features relevant to automated machine learning such as model compression and automated feature engineer. So we've seen that we've seen we how Hadoop can deal with hyper parameters and neuropathy a Search Search right, but that has to do with the training process. The feature engineering will be the step before all those begins right. You're, selecting your list of features, you're, maybe you're doing some kind of a feature selection with maybe some feature process, or maybe some transformation right.

A

You have to transform your your training data, so these are the things that we're constantly researching.

A

Ok, so how do you contribute? So you can find us on github and there will be a feel free to submit a feedback. Try our service and provide feedback feature request. There will be lots of Help Wanted features and a great way to contribute would be to help with the infrastructure and testing the improvements and maybe adding some new algorithms. So currently we support maybe made five or six different algorithms. So our intention is for cattle to involved into this open source platform for for the generic of automated machine learning right.

A

So we would really welcome contributions and there's an invitation to our slack channel. You cannot open it from here, but it's a move, you're interested okay. Finally, we want to thank a lot of our contributors. Some are in the audience today, so NTT high cloud IBM Cisco, have always been very, very helpful okay, so that is it for our presentation and we will take some questions. Yes,.

D

Can you change the DB back-end from the current implementation to something else, yeah.

B

So currently in v1, alpha 2, which is up come, which is a version which is coming in a month. So that's not possible, but in the 0.3 or in the next version the it will have a pluggable version where you could add any database where and you could just implement the common interface.

A

A

So the question is about the consumption of resources and Google what you mean really happy. So the.

B

A

Is about the conception of resources right so so resource you can configure on a per job level right so that there there are a few ways to configure how much are resource you want to consume. So one way is your budget right. So if you, if you want to have a smaller parallel budget, you can you can adjust how many twelves you want to run in parallel right and another way is to change this. The specifications on the training job, so some training jobs.

A

You can use multiple GPUs, maybe some you can either reduce numbers of you.

A

B

A

Do we still need machine learning experts after we have your architecture search.

A

So I think a lot of research going into how to efficiently generate new architecture search right. So it's not so a lot of this is still very early right, so we're still researching into how to efficiently generate how to generate.

A

Not not only in within the limited budget, but also, but also make it applicable to various kind of a problem, so I think currently, a new architecture search is pretty good with image recognition, but that there's there's a limited application elsewhere. So so this is so something that still requires research.

C

Thanks for the presentation, so I have a question for for this slice. Do you have do you have a simple use case? That's the ketchup and can show that ketchup and maybe in which stage that machine learning programmer can use it. For example, I wrote a simple training program in the gods way from some some model, but I don't satisfy with age, and so what's the euro euro cases that this ketchup product and 15 so I think I'm not quite clear of this festering. This problem.

C

May be a tactical use cases.

B

Yeah so so, as you described earlier, the wrong settings would actually have very bad performance of the model right so, for example, in the demo slide that he showed if he.

B

So here, if you see that for.

B

Certain parameters, you could see that the the final accuracy has come. What's the value here, just 11%, so it actually depends on what. How do you actually figure out? What is the best parameter? Hyper parameter values for your experiment right, so here the the clean advantage that you actually get. Is you don't need to worry about setting the hyper parameters? You basically get the best hyper parameters that is suited for your model, so, as you described earlier, the model performance continually improves, and finally, it gets to 98% in this. In this case,.

E

No question for Richard so from Google's perspective, it scubapro driven by gogo a I or Google Cloud.

E

Yeah internal Google, who is driving co, buffalo project, it's Google or Google, a I, so.

A

It is that cocoa organization is under the cloud AI organization, so the the the team that's driving. It is cool cloud.

E

So I kind of had a feeling that from googly eye, for example, tensorflow it's very hot in AI domain but looks like COBOL flow- is not that popular at this moment, maybe my time wrong. So maybe you can give me more insight about that.

A

So our our goal is to democratize AI, not just targeting tensorflow right, so we want to make it easy for people to build. End-To-End machine learning will float on kubernetes and it's just on the cloud. So that's why the ketchup and other frameworks are agnostic to framework okay.

E

Okay, thank you.

F

Rotten automatic, a machine learning Shanti, sir vomiting was a new sort of ladder. Macedonians one gentleman was rotini new feature. Second reward: we need to establish a new adventure. Our routine today share feature, even though she is here. Buddhist Roisin, Dhamma know what imposing a senior associate rather take a official 19.2.

A

Okay, so I think our time, maybe up I, do have a quick question.

A

Okay, oh thank you.