Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: MLOps at Snapchat: Continuous Machine Learning with Kubeflow & Spinnaker - Kevin Dela Rosa, Snap Inc

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

MLOps at Snapchat: Continuous Machine Learning with Kubeflow & Spinnaker - Kevin Dela Rosa, Snap Inc.

Training a machine learning model to support your use case can be difficult, but in actuality model creation is only the beginning. ML systems are complex and differ from traditional software systems; as such unique challenges arise when engineers or data scientists try to integrate and continuously operate ML systems in production. Applying best practices and principles from DevOps to machine learning systems (MLOps) can help practitioners navigate the entire ML lifecycle. In this talk, we will share our experience so far in applying MLOps to a computer vision use case at Snapchat. We will walkthrough the process of transforming a manual script driven process into a more robust and automated experience. We will describe our ML pipeline and how we leveraged Kubernetes, Kubeflow pipelines, and Spinnaker to achieve continuous integration, continuous delivery, and continuous training.

https://sched.co/ekC8

A

Hey everyone welcome to my talk over the next 20 minutes, or so I'm going to talk about emma lops, specifically I'm going to highlight how we automated various parts of our machine learning process using tools like kubeflow pipelines and spinnaker. But first let me hide my camera, so we can focus on the slides.

A

A quick note on me: my name is kevin delarosa and I'm a machine learning engineer at snapchat. I primarily focus on computer vision problems to support the scan feature in the snapchat app. That's my twitter handle feel free to reach out to me there or on other channels like linkedin.

A

So here's a quick overview of what we'll be covering first I'll. Give you a brief introduction to what I work on at snap and where machine learning fits in then we'll talk through the key components of a production machine learning system we'll touch on what ml ops is and then for the remainder of the talk, we'll walk through the various steps involved from going from an experiment to continuous machine learning in production.

A

Closing out with an overview of what this looks like for my team.

A

A

Started snapchat is the fastest way to share a moment at snap. We contribute to human progress by empowering people to express themselves, live in a moment, learn about the world and have fun together.

A

One of the ways we allow folks express themselves is through augmented reality experiences. We call lenses.

A

In the app these lenses can be unlocked directly through a snap code. Lens links and a lens explorer snapchatters can also discover these lenses using the scan feature by long pressing. The screen, while the camera is active, scan, brings up contextual lenses and tools that may be relevant to what the camera is seeing.

A

In this first example, we see a drink related lens come up after scanning a cocktail.

A

In the second example, we see a lens that was built with a marker template being unlocked by scanning a poster where markers are images like this poster that can be recognized and tracked by the snapchat camera to create 3d immersive experiences like the one shown.

A

Here, when you create these augmented uh reality, experiences in lens studio creators can specify scan triggers which tells snapchat what you'd like the lens to be related to in scan.

A

For example, if you add the sky uh scan trigger when submitting an astronaut lens like this one, uh in doing so, when a snapchatter points, their camera to sky and trigger scan, it may open the corresponding lens.

A

And, as previously mentioned, lenses built with the marker template can be unlocked in snapchat directly through the camera. Snapchatters, who encountered a marker in the wild, can long press on a camera to initiate a scan and unlock the corresponding lens.

A

Under the hood, scan is powered by machine learning. In both of these cases when scan is triggered, we use machine learning, approaches to automatically recommend and rank the most relevant lenses to show to the user.

A

And in production, if you followed the ml ops methodology, you end up with a system like this, which is a diagram from google cloud.

A

Some key components include continuous integration for building testing packet and packaging pipeline components. An automated machine learning pipeline for performing your data engineering model, training and model evaluation, workflows. Some feature store to feed data into your ml pipeline.

A

A prediction service powered by your actual machine learning model a continuous delivery pipeline for deploying your machine learning pipeline, as well as your prediction service and, lastly, some mechanism for monitoring performance.

A

As we progress through this presentation, I'll provide more details on each of these steps and how you can ultimately get there. Based on our experience.

A

But everyone has but but when everything's said and done, we all have to start from somewhere.

A

Machine learning is experimental by nature and the starting point for many new machine learning applications is often experimental code in jupiter notebooks like this one.

A

So there's obviously a gap between having a proof of concept, jupiter notebook and that big diagram. I showed you of a finely tuned and automated machine learning deployment.

A

For this we can use ml ops to guide us along the way.

A

Ml. Ops is a set of best practices to test deploy, manage and monitor ml models in real world production.

A

On the right hand, side you see some of the core concepts to keep in mind, while productionizing a machine learning system. In order to understand these you'll first need to recognize. There are key differences between traditional software systems and machine learning deployments.

A

For example, machine learning isn't just about code. Machine learning is really about the combination of code and data to produce artifacts called machine learning models and, in addition to code changes. One also has to be aware of data changes as these impact, the performance of your model and the reproducibility of your steps.

A

Additionally, the ecosystem surrounding machine learning deployments has a lot of moving parts that are not always seen in traditional server deployments.

A

For example, data pipelines, including etls, as well as data labeling pipelines, feature stores, as previously mentioned, and your actual machine learning pipeline, which can often be highly customized for a specific use case.

A

Now that's a lot of stuff to keep in mind. Instead of trying to do this in one big bang, you should instead opt to adopt a incremental process to gradually increase the level of automation in your system as things mature.

A

Generally speaking, these are the steps we used to reach continuous machine learning for the things like unlocking lenses and scan.

A

I'm not going to touch on the machine learning algorithms here, but more focus on the machine, learning, engineering and ml ops considerations, with the assumption that you have already done some experimentation on the ml modeling side.

A

So let's walk through each of these steps and what you gain along the way.

A

Firstly, you want to containerize all the things in this step. You need to break up your experimental jupiter notebooks into modular code and containerized programs, and once that is done, you add it to your source control system. Ie github.

A

After the step, changes to your algorithm and transform code will be now tracked.

A

You'll also have software that is easier to write unicode unit tests, for since your code is more modular and you'll more easily be able to reproduce things because you dockerized all the things.

A

Step two in this step: you want to automate your machine learning process for this. We use kubeflow pipelines.

A

For the uninitiated, this is what cube flow looks like in kubeflow. A pipeline is a graph describing a complete machine learning workflow. This graph specifies what components are on. How do you relate to each other and each component or step in the pipeline launches one or more kubernetes pods and acts like a function in that it has a name parameters, return values and a body ie docker containers in this scheme. Data is passed between parent components and children runs via serialized data.

A

Using kubeflow to represent your ml pipeline might result in a graph that looks something like this. First, you start off with some step that validates the input data to your pipeline.

A

Then we pass off to some component that extracts features from the raw data or the feature store.

A

Then we pass these features along to component that actually trains, a machine learning model.

A

After the model is trained and produced, we can do some model evaluation, like calculating accuracy, mean average precision, recall intersection over union or whatever what other metrics um over a validation set that are relevant to your use case then. Lastly, you'll typically do some model validation against some held out test data.

A

This can take the form of checking if the model performs at some minimally acceptable level of accuracy prior to uploading it to wherever you store or register your models.

A

Over the course of step two, you now have an automated way to kick off a run of your training pipeline to help enable reproducibility you can store metadata about each run and model.

A

You can track changes to your entire ml process by storing your kfp pipeline in source control.

A

You end up testing more things in the form of components like data and model validation, as well as unit tests over to model specification and integration tests over your entire ml pipeline.

A

You can also log model performance for each run to understand if your model is degrading over time or perhaps, if you goofed up on the algorithm side,.

A

Step three: now that your components are containerized and you have a pipeline that automates the ml process, we can move towards continuous integration.

A

In this step, you want to set up your favorite ci tool to help you build code and run unit and integration tests.

A

On my team, we use a mix of jenkins and drone to accomplish ci.

A

Either way you want to have some release step that publishes artifacts like docker images, which we publish to gcr google cloud registry or, if you're on aws, that would be ecr.

A

We compile versions of our kubeflow pipeline specification yamls and we version kubernetes configurations for the prediction service deployments.

A

In this step, you'll have unit and integration test automation at both the the component and pipeline level, as well as the artifacts required to deploy a server or training pipeline which will come into play in the next few steps.

A

In step, four, we start to talk about continuous delivery. In this step, you want to orchestrate the deployment of your ml pipeline using a continuous delivery system.

A

When you reach this stage, you'll trigger automated deployments of the ml pipeline. When your continuous integration system releases new artifacts, this will let you automatically train new models whenever the ml process changes.

A

Additionally, if you use a cd system, you'll have repeatable deployments for your ml pipeline and also have the ability to do things like roll back to previous versions straight from the ui. So for this step we used spinnaker.

A

For the uninitiated, this is what spinnaker looks like spinnaker is a cloud native, continuous delivery system, which makes your deployments fast, safe and repeatable.

A

In spinnaker, you define one or more pipelines to manage the deployments which consist of a set of stages.

A

A stage is an action for the spinnaker pipeline to perform like deploying a manifest running, a job, kicking off a sub pipeline, resizing a server group and so on and so forth.

A

Deploying our ml pipeline might look something like this. Your initial configuration expects a set of artifacts as input.

A

In our case, this would contain the kubeflow pipeline yaml artifact and some configuration to help us pass in proper parameters when invoking the kubeflow pipeline, then next you might have some smoke test to do non-excessive tests against your new ml pipeline, but this is optional and really up to you.

A

Next is the main part of this pipeline running the ml pipeline stage, which does the actual, deploying or submission of this pipeline.

A

For us, this looked like a small kubernetes job that first submits a run of the ml pipeline to the kubeflow cluster waits for successful completion.

A

Then parses the output to produce a spinnaker artifact that can be passed around in subsequent stages in the spinnaker pipeline.

A

Lastly, if you have a model server deployment, spinnaker pipeline, you can invoke that step directly at the end of this pipeline, though, at this point in your project life cycle, you probably don't have that yet more on this in step,.

A

A

in step five, we achieve continuous training to do this. You update your cd pipeline to trigger deployment of your ml pipeline, either on some set on some set schedule like daily, weekly, monthly or upon availability of new data.

A

The route you take here will depend on your use case and availability of new data.

A

We do this in spinnaker by using built-in triggering mechanisms like chrome jobs and pub sub message triggering after completing this you'll automate the process of training, new models upon the availability of new data.

A

In step six, we achieve continuous delivery for the model server or prediction service.

A

We did this in spinnaker by setting up a pipeline which specifies how to deploy the model server and triggered on changes to the server configuration via continuous integration or upon the availability of new model artifacts created by the continuous training spinnaker pipeline discussed in the previous steps.

A

After this stage is completed, you benefit from the continuous deployment of new model servers and also make gains with respect to reproducibility.

A

The spinnaker pipeline here should look similar to what you'd expect for traditional server deployment, so I won't go into great detail on this slide. You see a sample spinnaker deployment that deploys a canary waits for manual. Approval then deploys to some production environment.

A

Lastly, you'll want to invest in monitoring in this step. You want to make sure that you have the proper telemetry in place for your server.

A

Additionally, you'll need mechanisms to detect deviations in model performance for the former, you have the usual suspects of latency traffic errors and saturation for your model performance. If it is hard to get direct feedback on your model's predictive accuracy, you can use proxy metrics that track the results of acting on your predictions.

A

For example, here you might measure if there are shifts in click-through rates.

A

After this step, you'll unlock the capability to determine if your model or ml process needs to be changed.

A

So cool those were the seven steps we took to reach quote unquote: production grade machine learning.

A

At this point you will have achieved a new, continuous thing, continuous machine learning.

A

This diagram summarizes the journey we took from ml experiments in jupiter notebooks to continue integration, using tools like jenkins, continuous training, using kubeflow to automate your ml process continuous deployment through spinnaker to monitor to monitoring metrics in your kubernetes pods, which will then ultimately lead you back to doing more experiments and refining your process.

A

Here are some useful links for those who want to learn more about snapchat ml, ops in general or the various tools I mentioned I'll stay here for a few seconds for those who want to screenshot this for future.

A

Reference, okay anyway, thanks for listening.

A

Again, I'm kevin dela rosa signing off have a great rest of your day. Thank you.