Cloud Native Computing Foundation Kubernetes AI Day North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Case Study: Developing and Scaling Kubeflow’s Web Apps - Andrey Velichkevich & Kimonas Sotirchos

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Case Study: Developing and Scaling Kubeflow’s Web Apps - Andrey Velichkevich, Cisco & Kimonas Sotirchos, Arrikto

At this moment, Kubeflow maintains at least 5 different web apps, for managing Notebooks, PVCs, Tensorboards, Models, AutoML Experiments, that allow users to interact with the platform. At their core these web apps act as a graphical interface for performing CRUD operations on top of K8s Objects and Custom Resources. Designing, creating, and maintaining these apps is not a trivial task. In this talk, attendees will learn how the Kubeflow community overcame all the challenges to create true cloud native web apps, for managing ML workflows on top of K8s. Follow our journey as we explore the architectural decisions we made regarding authentication with Istio and authorization with K8s SubjectAccessReviews. How we factored out the common code and enabled application scalability. The UX decisions for managing K8s objects via a GUI. And last but not least, how we can efficiently fetch new data for Kubeflow dashboard in the context of how users can perform advanced AutoML techniques.

A

Hello, everyone, I'm kimonosotircos and I'm a software engineer at ericto and also a lead at the notebooks and manifest working group and also the release manager for kubeflow 1.4.

B

And I'm andre veliskevich, I'm a social here in cisco and I'm the elite of autumnal and training working group in cuba.

A

And we're really excited that we are part of the kubernetes ai day and thank you for joining our talk and today we're going to talk about how we went around developing and scaling our keyflows web applications. But before we start actually discussing about the applications, we'll need to learn a little bit more about the users and in our case it's an entire ml team with which will have different roles that will need to play together to have an efficient workflow. We'll have the data engineers that will be handling the initial states of the data.

A

We'll have data scientists as well as ml engineers, will be taking this data analyzing, creating the models, training, the distribution in a distributed fashion, and then we will also have devops because we're kubernetes, who will be helping along the process and also taking care of model serving and all of this will need to be able to happen in one cohesive platform, with some intuitive uis and in this presentation we're going to see how did we go around with q flow to build such a platform that will have these three pillars, which is team, isolation, scalability and user experience.

A

So the first one is team isolation, and since we are building a platform on top of kubernetes, the solution here is: we only rely in namespace isolation by which means that different users and teams will have their own namespaces and they will be using our bug, which is kubernetes native to control who what entities will have access to which resources.

A

But let's take it one step higher and see. How can it work from a user's perspective when they're at their browser, a user, will just send a request? For example, they would like to delete a notebook to a web application living in the keyframe namespace, and then the web application will perform an action like delete the actual notebook on behalf of the user, with the permission it has and with its own service account.

A

But here there comes the first initial question, which is: how can I ensure there's name space isolation and our backup is being respected if the web app has permission and can do anything, it wants well. The answer here is with subject access reviews.

A

Subject: access reviews the same mechanism that kubectl oauth can I is using under the hood in which we can ask kubernetes if a user has permissions to make an action on top of an object in the cluster, so the backend can create the subject, access review and be sure that the user is indeed authorized by rbot to do the action they want. But this also then raises the next question, which is okay? How can the web app know which user is actually connected and doing the request?

A

That's the answer here is with http headers. The web apps will expect that a specific authentication mechanism where this is, for example, google iap, will be in front of the cluster and ensuring that there is an authenticated user and setting the setting this user identity into specific headers and to get this all together.

A

We can also do it by complete open source solution by using dex, so a user will be connected to istio's ingress gateway, which will be working alongside dx4ydc and ensuring an authenticated user is doing all of the actions with the web apps in and by respecting rbac. So the three things we need you need to remember here is that we have namespace isolations for users and teams. We have always have authenticated users and we are using subject access reviews to ensure we comply with kubernetes are back.

A

The next part is scalability. Well the cheat solution. The cheat solution here since we're at kubecon is we're allowed kubernetes for a lot of scalability, for example, and this has to do both for the actual ml workflows as well as the web, the web applications.

A

If we have a lot of requests, then we just spin up more pods for our web apps, which are restful, but at this point I'd like to also discuss the scalability point of view from the web apps and what we have to do to actually scale our web apps when we have hundreds of requests coming from hundreds of users that want to do things on top of thousands of kubernetes objects. So how can we do this efficiently, I'll just focus on?

A

How can we actually keep on fetching the data and have an auto refresh mechanism here and what we currently do is exponential polling?

A

So we ask for new data every one, two four eight seconds as long as the data is still, but once we have new data, then we reset the counter and start asking more frequently, but this has some drawbacks, like the user might wait even 32 seconds if there is no activity on the on the resources on the list of objects or on the other hand, if there's a sudden change, then all the hundreds of users will be asking for the same data with like consecutive requests, which can result in a lot of load.

A

One future improvement we've considered here is to keep the simple polling mechanism ev every a request, every four seconds, for example. But the request will be a simple head request and will it will be getting only the resource version of the latest list object in kubernetes, and only once the resource version is changed.

A

Then then the ui will do a get request to get the latest data and we can even do a slightly better optimization here, because even in that case, we might be still getting thousands of objects in one request, but we can actually use kubernetes api chunking to cut these requests into smaller pieces so that the load is less on the on all of the components and we can have data more fastly to the browser.

A

We've also evaluated other solutions like web sockets or server-side events, but since we were a little bit tight on time, we can discuss more on this on our q a session and last but not least, is the user experience part and it's really important for us, because in q flow our targeted users, while they're working on top of kubernetes, they do not have experience of kubernetes or we cannot assume that they have so our users.

A

The email scientists, only know how to do their specific ml tasks and use their algorithms, but not, but we cannot expect them to be able to ctl logs to see the logs or could still describe if something goes wrong.

A

In order and be able to debug it, so our web applications will need to be able to provide all of this information in a user-friendly way, while at the same time layering this information, so that the more machine learning information is exposed to the user first and the more advanced kubernetes concepts are also shown but kept for an advanced user.

A

But because I can just describe a ux with words, um you will need to actually see what we talk about and in order to see it. You'll also need to understand what are the requirements of an ml scientist and what do they actually need from such a platform and further understand this uh andre is going to walk you through. How can uh and what does an ml scientist do and specifically, how can they do automl and how can they do it on top of kubernetes and with kubeflow?

A

So andre back to you.

B

Thank you camonis. uh Yes, so I'm going to tell you a little bit about uh what is autumnal and how do we solve this problem in qfl? So optimal is the process of automation, machine learning tasks, and especially here in this page, you can see the landscape for outsmall, where users can provide the training data in the configuration space similar to the search space description.

B

A search space can be differently depends on your task, so it can be architecture, it can be hyper parameters, it can be features and all this configuration you can pass the optimizers based on an algorithm, for example, based optimization or neural architecture, research, algorithms and then optimizer provides a configuration to you.

B

Then we passing this configuration to the model with the test data and pushes model to production uh covering ultima, covers a lot of different aspects such as feature engineering model, compression neural architecture, search, hyper parameter tuning, and we try to adopt all of these features in cloud native way in queue flow. So how do we solve this?

B

In q4, we have a project called cadib, which is kubernetes native, open source platform for automl and this project, including keep flow distribution, so which is a very good part of this project, since it is built on top of the kubernetes, it is agnostic to machine learning, frameworks and programming languages, so you can tune your hyper parameters in any languages and also we have in place support for any sort of the kubernetes crd, so cadib can be used as the orchestrator on top of your kubernetes custom resources.

B

Also, we support famous optimization framework such as hyperopt, optuna, skopt and many more we're continually evaluating with supporting new algorithms and also our researchers can use our platform to develop and evaluate new ultima algorithms. Since this platform is uh language agnostic and also since it built on up of kubernetes, uh we can deploy kdp on local uh or public or private cloud. Also, we have a native integration with key flow components such as notebooks pipelines and training operators going to k-dip architecture.

B

It's quite straightforward: since k-dip has the kubernetes customer source, we have three different controllers: one is uh experiment controller, which is eventually takes the user experiment yaml and try to proceed this yaml uh parse the information from the user needs. Then we have a suggestion, controller, which has received the search space uh description for an experiment and also algorithm and creates the algorithm service when it's eventually the optimal algorithm is running.

B

Our ultima algorithm can produce the hyper parameters or renewal architecture based on the needs, and then we have a trial controller trial control just spawns the trials where the actual training job is running and since trial is an obstruction can be any so any type of training job.

B

We can define even the taxon pipeline or even the argo workflow in our trials, and then we, uh our caddy, is able to actually parse the training results from the trials, and then we send this metrics to the db and then the we send the evaluation matrix back to the suggestion service and the produce new hyper parameters.

B

Speaking about the tip one of the experiment, one one of the example of our experiment: here you can see the yama structure, so our users define the experiment budget with the specific trial information. Then they define objective. uh Then they define an algorithm and then identify the search space. We will see the more information, the demo that I will show you later and here you can see the child template and since, as I said before, trial can be anything uh you can define your specific specification here for your customer source.

B

uh You can define the whole argo workflow here and you can set up the your experiment with the preprocessing step with post-processing step, so it's very powerful if you want to have like more very like some uh sophisticated, hyper brand new tuning examples, and with this with that said, I will jump directly to our demo that we show all of the ui that we built in queue flow. That cameras mentioned before and I will just uh show how autumnal is working in uh keyflow. So this is the kubeflow central dashboard.

B

I think all of you familiar with this analytic dashboard, because we uh trying to uh to show every time our our ui and here uh the devops engineers can always like see the cpu utilization, see the post. Cp utilization see all of the uh monitoring information that they want to. uh They want to reach from the data scientist team.

B

Of course, ui is split across the name space, so you can as common as mentioned before, you can easily uh adopt this ui for your huge ml team uh with the namespace isolation and everything will be work. So uh here you can see, you can see the notebooks ui where you can define the notebook. You can choose the images from the predefined images.

B

You can also define the cpu gpu, the volume where you can, when you want to take your data and in the meantime, once you create a notebook, you can click the connect and you will be in the familiar jupiter lab environment, which is everything built on top of the cloud. So this is very powerful because you can run the predictions you can run the pipelines with your notebook. You can run the training, operators or kdf experiment, then you can also create the tester board using our keyflow ui.

B

uh This tensorboard also very powerful tool for the data scientists to be able to develop uh algorithms uh in cloud native way, and you can also deploy your own tensorboards. We can have also like ui for the volumes where you can monitor the volumes and also we have the pipelines ui, uh which is basically the queue flow pipelines project, and also here you can see all of your pipelines and pipeline runs uh recurrent, runs and all of the functionality from the kfp.

B

And then, if we jump to kdp experiment, I will try to submit new experiment to show how those work from the from the ui perspective. So, first of all, we just need to define the name of the experiments. We will use the base optimization for this demo. uh This is the budget that I mentioned before. So you can define the the number of trials that you want to they want to run.

B

Then you define the objective with your metrics that you want to tune. You can also define the all of the metrics that you want to uh also parse for the training containers. Then you select the search algorithm, so we have a lot of algorithms that k-dip supports. uh We will choose the basic optimization for this demo, but you can also choose the list of the algorithms from the nas as well.

B

uh So let's just select the base optimization here you can see the algorithm settings uh which also user can modify and adopt for their needs. Also, this is the aerostomic section uh which can help you to avoid overfeeding and very powerful, and you definitely need to use this in your hyperparameter tuning experiment but validity experiment. We just simply run one algorithm and this is the search space. So in this example, we will tune learning, create a number number of players and optimizer.

B

Also, of course, you can add the new parameters uh with the particular distribution we support, like integer, double discrete categorical parameters. You can edit the current parameter, you can delete them um and you can define like your search space for the experiment for the matrix collector we support. This is the list of the supported matrix collector from the kd where we lose std out, and this is the child template. So eventually we will use the default settings, so we are not going to change the any uh any information from here.

B

This is a template, that's where our training process uh will be run and we are going to tune the learning, crate number of layers and the optimizer. So we just need to define the reference for the particular parameters and for the advanced user we have a button which just produced the whole kubernetes yama structure. Where you can define the object metadata, you can define some api's parameters which just the forms does not support and uh which these provide like more functionality for the uh more like this uh native way.

B

So once this experiment is running, you can see that some of the experiments uh have been created before you can see the time when it was created, you can see the number of successful trials the optimal trials. Also, you will see the parameters that the best trial produce and if we click to the particular example, you will see this plot, which actually introduced all of the all the distribution between different hyper parameters, all of the results and these parameters achieved and also here at the bottom.

B

You will see the overview of the experiment with this, with the current status, with the best trial with the child parameters uh and also the current experiment conditions uh that experiment is met. If you click to the trials page, you will see that that all the trials that a current experiment uh was uh was running and if you click to the particular trial, we will see the distributions.

B

uh Sorry, the the metrics, that our uh the metrics that they just collected and you can always like monitor, what's happening with with your metrics in a particular trial and also which is very powerful. We highlight the best trial and you can see the results with this bad child by clicking on the name. Of course, we have like more the uh more discreet details with uh with with the experiment, so it's like, for example, like objective trials parameters and, uh as I said before, for the kubernetes for kubernetes user.

B

They can always see the java structure for the experiments uh with all in the required information. I think our experiment is running. We we're not going to wait, but here you can see in the lifetime how the parameters will be changed will be changing and also you can see which trials are currently running and which trials was uh were succeeded so and uh last, but not the least.

B

I just want to briefly show the models ui, which is related to kf serving- and here I I already like- deployed one inference to show which, like kind of the parameters you will see, so this is the one inference that I deployed before and if we click to this inference, you will also see the overview of the inference of the current statuses of this inference. You can see the details of this uh keep serving uh instance. You can see the locks from the key serving and you can, which is like very powerful.

B

You can see the metrics, so this metric is produced uh by grafana and we can see what is the current status for the for our like uh inference from the kf starting site and, for example, uh if I try to if I try to invoke this service with some of the predictions uh yeah. This is just a simple. I missed example, and I try to invoke this service with the predictions. We will see that the metrics uh should be actually changed, and so we will see the information.

B

That's uh that's actually a grafana per use, so yeah as we can see here, the request has been come and we will see like the uh the the devops engineers can just monitor this page.

B

uh By doing this analysis- and with that said, uh if, if you have any questions regarding the demo, we'll just feel free to just answer in the q a session, because we don't have much time right now, but I will jump back to our presentation and uh and the last, but a very important thing that I really want to mention about the our community, uh we're growing very fast. We have more than 12 thousand commits and two uh twenty thousand github stars on kubeflow, which is very exciting and where we we want to say.

B

Thank you so much for contributions um as uh me and camonis were part of the autumnal and notebook working group uh really want to. uh We want from you to attend our working group meetings. We have our like regular meetings every week. Please join our slack channels that we that the information here, uh if you using keep in production, please update the adopters list. This will be very available for us to to make the connection with the users. Also, please feel free to check our latest presentations with the demos and with our.

B

Latest summits, also, if you want to contribute, uh please feel free to follow the developer guide, the help wanted issue and submit the new proposals and which is very important, please feel free to check our mobs for autumnal and for qfloo in 2021 and with this said, thank you so much for listening us today. We're happy to answer all of your questions.