Cloud Native Computing Foundation Kubernetes AI Day NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Building Enterprise AI/ML Platform on Kubernetes - Maulin Patel, Group Product Manager

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Keynote: Building Enterprise AI/ML Platform on Kubernetes - Maulin Patel, Group Product Manager, Google

AI/ML practitioners are increasingly leveraging Kubernetes for data processing, training and inference needs. Kubernetes is well suited to address their needs given its support for auto-provisioning, auto-scaling and various machine types (e.g. CPU, GPU, TPU). AI/ML practitioners also benefit from Kubernetes dynamic scheduling, high availability, job API, portability, customizability and fault tolerance capabilities. As such many organizations are standardizing their AI/ML Platform on Kubernetes. In this talk, we will share the best practices for building AI/ML platforms on Kubernetes.

A

Thank you for kind introduction. I'm. All in I'm, the group product manager on Google kubernetes engine world is moving towards Cloud native Computing and kubernetes has become the de facto standard for cloud native computing. Many organizations around the world are building machine learning platform on kubernetes, since there are many commercially available machine learning platforms that are fully managed.

A

One would wonder why do organizations build their own platform on kubernetes, so we interviewed many organizations in order to find out their motivations and also to learn best practices, and we found that there are four key reasons why organizations build their own machine learning platform on kubernetes.

A

The first and foremost, is the portability, so organizations design their platform, often for multi-cloud and hybrid architecture. They want the freedom to write the code once and run everywhere, including on-prem and in a multi-cloud environment. As such, they prefer open source, software-based stack. The second key reason is customizability organizations that have a deep kubernetes expertise.

A

They like to customize their stack and they appreciate the flexibility to do so. They also want granular control over the resources.

A

Third key reason: why organization build their own platform is the performance, so many organizations which are deeply Savvy actually like to hyper optimize, their architecture to fine-tune Performance and save cost. Often they are also motivated by unique security and networking requirement. The fourth key reason is the consistency. Very often organizations want consistency between their machine learning operations and devops. They don't want them to be siled.

A

They don't want diverging architectures, that's one of the key reasons as well, so when we think about developing machine learning and training as the key requirement, when you think about building an end-to-end machine learning applications, you typically think about models and training.

A

On the other hand, in order to build an end-to-end machine learning applications, you have to accomplish many complex tasks. This includes Gathering the data cleaning, the data extracting the features, building the models, provisioning resources to train the model and when model is trained, putting in a production for serving the prediction and then monitoring the model.

A

All of this requires a very sophisticated machine learning platform and tooling, which can make the entire end-to-end process simple, easy, reliable, automated and reproducible.

A

So machine learning is inherently an experimental discipline data scientists have to gather the data and extract the features and try out different features. Try out different algorithms, do hyper, parameter, optimizations and many different things in order to build a model and then compare many models across for performance comparison and they want to track what worked and what didn't- and you have to do this in a very reproducible and reusable way, and there are four key technological building blocks that enable us to do the prototyping and experimentation in an expedited way.

A

So what are those three balding blocks? So the first and foremost is a feature store. What is a feature store feature store, is a centralized feature repository which basically enables organizations ml practitioners to discover and reuse features.

A

It also is a scalable architecture which serves features for training and prediction in a low latency way. It also helps to alleviate training serving skus, and there is a very popular open source feature store, called fist, which is widely used with kubernetes. The second key building block is the notebooks. Notebooks are the gateway to data science.

A

Notebooks are also integral part of development, training and deployment. Workflows. Jupiter is one of the very popular open source notebook. The third key requirement is experimental, training and visualization, so data scientists want to analyze and compare different training runs and tensorboard here is one of the very popular open source visualization tool.

A

The fourth one is Pipelines pipelines are the way to automate workflows for efficiency and Rapid. Iteration Google pipeline is a very popular open source pipeline that we have seen many of our customers use with kubernetes.

A

So another key aspect is constant retraining, so world is always changing and data moves fast models get stale, so data scientists have to constantly use the new training data and new algorithms to build the new models, and we want a platform that can make this simple, easy and cost effective. This is exactly where the kubernetes showing kubernetes offers unmatched scalability. You can scale up your training environment from a single node to thousands of node, and this is an ideal platform for distributed training and Hyper parameter.

A

Optimizations open source School flow training operator is a very popular operator which implements a major machine learning framework and helps you do distributed training. The second key requirement is the performance, and here again, kubernetes is quite helpful. It actually natively supports Hardware accelerators such as gpus and gpus, which can help you save time and money by Expediting. The training, however gpus and tpus alone are not sufficient to get the best performance. You also need High, throughput, networking and storage as well as high iops, in order for you to keep your gpus and tpus busy.

A

Last but not least, one of the key requirements that do it yourself, machine learning platform developers have is the cost efficiency and they heavily used discounted spot VMS in order to minimize the cost. However, as we all know, spark VMS can be preempted so remember to checkpoint, frequently or use elastic training operators which do this automatically. For you.

A

We all know gpus and tpus are very expensive resource and getting the highest utilization possible is one of the key design consideration for ML platform builders mammal platform. Businesses also use batch more for training, as badge gives them time and flexibility, which also saves you money.

A

So after your model is trained, you want to put it in a production with a live data. To do the prediction, and in this case you have to enable logging and monitoring which can alert you when things go wrong or slos are violated, and this also has to be done in a way that is wetted, auditable and reversible.

A

So kubernetes is one of the best way you can actually optimize your serving cost. The reason it is is because it allows you to Auto scale from zero node to thousands of nodes depending on the traffic. You can also do scalable multi-model serving in order to tightly bin pack. Multiple models to reduce your cost caser is one of the popular open source stack that runs on kubernetes and helps you achieve many of those properties.

A

Another key requirement is model performance monitoring, so kubernetes observability Primitives are typically designed to help you monitor uptime, latency, cput utilization memory, utilization and other service level indicators. They are not well suited for machine learning model monitoring. For several reasons, machine learning, practitioners, they really care about accuracy, precision, F1 scores and data drift and concept drift. They also really care about training and serving skew KL Divergence feature importance, and things like that. Last but not least, model fairness is also an important consideration and good news here.

A

Good news here is that there is an open source framework called evidently which can do all of this, so you can use it if you are designing an open source based machine Learning stock.

A

The last but not least, upgrades are very important part of machine learning life cycle, and here the platform Builders typically allow you building blocks that can do a B's kind of testing, so you can test your models with different demographics before they go into production, so the model management and governance consideration grow across the entire machine learning operations workflow from end to end Model Management deals with registering your models.

A

Collecting the metadata, also doing version management, governance of the model deals with issues of reviewing the model approving for production and when things go wrong going to the rollback. Here again, there are very nice open source tool. Machine ml flow, which can be used for Model Management.

A

Another key requirement is metadata and linear, striking for governance and compliance. We want to automatically track inputs and outputs of all your components in the pipeline in order for us to create a lineage graph. So the all the dependencies that produce a particular Run for the model can be tracked and visualized.

A

We also need the auditability capability for model provenance and compliance. Reason. Last but not least, explain enable AI is a very, very important in order for us to build the trust in the model and also to assess whether model is fit for the purpose.

A

So those of you who are considering building machine learning platform natively on kubernetes, uh one of the great places to start is crew flow and I'm very pleased to announce that Google has decided to donate cook flow to cncf. So all of us together can innovate and bring it to the next level.

A

So machine learning and AI are changing the way we work in an unprecedented way and may the AI force be with you. Thank you.

B

Brilliant um yeah. Thank you very much. um Yeah awesome. If you didn't see that news, that's really cool right that kubeflow joining the cncf I, think you know it's! It's nice! It's gonna, be official part of everything that you know we're all doing this week, um so yeah. Thank you very much. I! Don't think we have any time for questions. We need to move straight on to the next talk, but thank you again.