Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 16 Jul 2019

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: The State of AI and Machine Learning Operations in 2019 - Alejandro Saucedo

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

The State of AI and Machine Learning Operations in 2019 - Alejandro Saucedo, The Institute for Ethical AI & Machine Learning

This talk will provide key insights on the learnings I have obtained throughout my career building and deploying machine learning systems, together with some of the technologies available in the open source "Awesome DataOps / Machine Learning Operations" list. I will be mainly covering the open source tools and frameworks available in 2019 to help you expand your DataOps and MLOps infrastructure. This talk will cover the technologies available to support specifically around reproducibility, monitoring, compliance and orchestration of data and computations. The main technologies that will be covered include PMML, Data Version Control (DVC), ModelDB, Pachyderm and Seldon.

https://sched.co/Nrw1

A

Thank you again we're gonna dive into the state of production, machine learning in 2019 my name is Alejandro Salcedo. You can find my Twitter down there. I am the chief scientist at the Institute for ethically ion machine learning and I'm. Also the engineering director at this open source company called Selden technologies.

A

The Institute is a research center that focuses on building standards for the use of machine learning in production and Selden is an open source, machine learning, deployment and orchestration platform that allows you to deploy your models from research to production and you can check out and try out the platform with Selden on Selden delaio. So let's talk about data science projects, so small data science projects. In essence, they are not not a big problem, so you tend to have two workflows. The first one is model development and the second one is serving your developed model.

A

You know it consists of the usual steps of you know getting some data and cleaning it defining some features to transform the data, selecting a model defined in scoring metrics and then iterating once you're happy. You persist that model and then it goes to production to be able to serve any inference data that you see are coming through right. So in small projects it works relatively well. However, as our data science requirements grow, we face new issues, increasing complexities of flow of data, so who here has used crontab? Let's see a show of hands?

A

Yes, so people may know what crontab nightmare looks like when you scale to much without the process around it. Each data scientist has their own set of tools so I'm, like tensorflow, our spark. You know and good luck taking that you know away when, when it's actually a very productive to serving models becomes increasingly harder, as there's a larger number of them in production.

A

There's, there's more high parameters and in place, there's different versions of each model, running with different train data, etc, etc, and when stuff goes wrong, it's hard to trace back, not only in terms of the people that are involved, but also the responsibility that is laid. So how do you manage who's whose person needs to be in charge of what and then, as your technical function grows?

A

Also, should your infrastructure- and that is where the concept of machine learning operations comes in, so it is this layer of complexity that is not just product or features, it's also processes and best practices, and it is challenging the role of machine learning. Engineering is still being defined. It is the intersection of software engineering, data, science, DevOps, etc, and not only that, but when you see a job description for machine learning engineer, it tends to be something like we need a professional with a PhD.

A

You know five years experience software development experience with DevOps tools like Jenkins. You know data science experience to build models, consultant experience to figure out where to apply them, so a McKinsey consultant for a salary of an intern right. That's what you tend to see in this sort of stuff, so it is it is. It is currently being defined because it's a spectrum and we're gonna cover some of that and we've also mapped the ecosystem. We have another link that cannot be reached, but it's still in the same repo github.com slash ethical ml.

A

This has one of the largest and longest toolset to be able to find frameworks to manage your production machine learning. If you see a tool that you can't see there, please feel free to contribute the principles that we're going to cover today, however, are the principles of reproducibility orchestration and explain ability. So, let's get started with reproducibility reproducibility encompasses the concepts of model and data versioning so being able to keep track of your models as they're in production, but also in development as you train them.

A

This requires an abstraction of the individual steps into atomic self-sustaining and versioned models, and it includes the data that comes the code, slash configuration and the data that comes out, and this is complex not only because you already have the complete the challenge of keeping track of these three three different concepts, but also that it goes one level higher into the definition of your computational pipeline and you may have some feature generators. You may have some some models, some some data cleaners that you may want to use and reuse right.

A

So you need to keep track of what what is inside, but how it? How it all stitches together and version that in in some cases as well, so what we're going to dive now into is a a case study an example of reusable pipelines in an NLP use case. So we're going to be using this technology that were contributing to you can also you know, access it through this link or it's on the gate, repo which we're going to link.

A

But this is basically what cube flow is it allows you to define computational pipelines of your machine learning, and the interesting thing about this is that we have reusable steps of an NLP pipeline. What you see here is an NLP classifier. It basically takes an SL, a sentence and it states whether it is it should be moderated or not it's hateful speech or not, and this is basically something that is done in forums. We took the reddit hate speech dataset, which is basically the data set.

A

That moderators was collected from moderators, that removed comments, and what we show here is we have a continuous stream of tweets, and these tweets are the brexit tweets. I, don't know if you're familiar with breaks it in the UK, but it's basically processing all of brexit and what we're gonna do here is we're gonna train another pipeline where the last step is a deployment you know using Seldon, so we're using cube flow to run that our loading of the data, the cleaning of the text, the tokenization, the vectorization, the prediction etc.

A

So here you can see also in the logs the input and output. I think the tokenizer shows it a bit better. So here you can see this the string and the tokens coming out right, so you can see through the UI and ability to train pipelines, so we're gonna go here. We're gonna select the pipeline that we define programmatically and you know we're gonna create a trigger.

A

So there's gonna be row number four here we can define some high parameters in this case we're giving our data set for the github comments, but we can also have a Twitter data set of like Arno sentiment analysis. So as long as you pass a CSV file together with you know the name of the column where you have the features, the text and the name of the labels that you're gonna use to train your model right.

A

You can also pass things like the number of tf-idf vectors, so the the Engram range and some hyper parameters for the different steps, and then we can actually just trigger it from the front-end. We can just basically see the running pipeline in real time, so this is actually running in my local machine and, as you know, you know we don't have the best internet in this place at this point, so it's gonna probably take a while the first step.

A

What it did is it created a persistent value, a persistent volume claim and it's going to attach it to every single container.

A

So the containers can use that volume to pass the data from one to the other step to the other step, and this is interesting because there's full provenance of the data that is currently available and what that means is that every single R on the inputs and outputs of every single one of those steps is actually stored in a persistent volume that then can be backed up right and then the internet right now, it's yeah, it's like connection refused, so I probably need.

A

You know that the VPN but I actually have it like running on an online cluster that I'm going to show you, because I was preparing for for this just in case and I'm actually going to also connect to my server just in case as well, but yeah. So basically, what you're doing here is you're training a pipeline and the end end result will be a basically a run that you're storing with with a respective persistent volume.

A

So in this case you know it downloads, it cleans it etcetera, etc, and everything is stored in this persistent volume, with the ultimate goal of deploying it into your kubernetes cluster. So that is, that is the whole essence of this, of this multiple steps that are involved there, and you know there are also several other tools that you can use for the reproducibility of your machine. Learning such as you know this tool called beta version control. This is I, get hub, get like fork.

A

That also allows you to take to save state of all of the different components they emit the data, the code and the inputs and outputs, etc. So the way that you interact with it is very similar to the way that you interact with kit and they have a very, very interesting approach. Ml flow is another tool from data bricks. They have an approach to manage your experiments and track your experiments and then finally, we have another tool.

A

Called pachyderm pachyderm focuses on fully reproducible pipelines, so there is actually a very interesting integration that we that we do with with with pachyderm as well, and that is for this for the piece of reproducibility. This is an interesting concept, because all the all the different parts of the grid that you saw you know I could dive for two hours and talk about them right, so they are quite complex, but they're in the important thing from this talk is to get a feel of the ecosystem and to more than anything, just get involved and help.

A

The conversation help us push the conversation forward now. The second part is on model orchestration. This involves training and serving models at scale, and this is basically what we do primarily at Seldon. We, we have an open source framework that allows you to containerize language agnostic models. This is this can be Python, of course, but also can be Java. Are you know if you're feeling, you know like a cowboy?

A

You can also the Fortran I mean if you build a wrapper for it, I, don't know why you would, but you also can and cap so late legacy models, and this involves also about serving at scale. So it's not just about the complexity of putting the models in production, but it's about also orchestrating them. Standardizing the metrics and we're gonna talk about that stuff so, and this is a hard problem because it becomes a computational resource allocation right, its services with different computational requirements.

A

You know with often complex computational graphs, and you know we need to allocate the right resources at the right time to also not overshoot right, and this is a hard problem. Fortunately, you know we have managed to leverage a lot of the kubernetes infrastructure and, with the you know, recent relatively recent debut of CR DS, for you know the last two years, we're also able to abstract some of these complex concepts of infrastructure and being able to define them, define them in encapsulated components.

A

So then, we can suggest design patterns in the same way that you would in object-oriented programming right and we're gonna actually cover some object-oriented design patterns for your infrastructure, as we discuss this, and the reason why also serving an unemployment of models is hard is because it introduces a layer of governance and compliance so that already ambiguous field of machine learning. Engineering that we touched upon is in the intersection with other industry, specific fields like policy, expertise, industry domain experts, and then that forms the centerpiece, which is the the question of what are the standards.

A

You know what what are they? The key processes? Metrics. You know a framework risk assessment processes that we can have in place to make sure that we have our production machine learning following the right policies, etc, etc, and we're actually contributing to several algorithms, with I Triple E and several of the other bodies like ISO to define standards like the algorithmic bias standard, and in order for us to be able to go one level higher, we need to define the standardization of metrics right. We've done this for microservices.

A

We know that latency is something that you would track. We know that requests per second errors, but when it comes to, when it comes to models, you have other metrics that you may have to standardize like accuracy, precision recall model divergence, etc, etc, and what is interesting from from there is that, then, is mapping those metrics with your internal policies to create this concept.

A

We call programmatic governance right, not only you define your policies, you know with with actual writing, but you're able to also encode them in your impact and code them through your infrastructure right and in this case, what we provide is some you know, standardized metrics with any model that you deploy in production, but we are currently working with the cube flow team, and this is a very interesting initiative to create some standard for for those metrics that are going to be exposed so that any met any any model that is deployed in production.

A

It's not always going to have them, but we can all have at least an agreement that at least it should- and that is that is an important thing. The second piece is also standardization of errors and logging right right now, a lot of the times it is data scientists having to maintain their models in production, but with the introduction of these tools, you're able to create a division so that there is a distribution of responsibilities there.

A

The scientists, the data science, sis admins, do sysadmin right and you can also then go into higher-level tasks like complex deployment strategies. So with Selden you can do things like explanations, lie, detection, feature, transformations, etc, but this is about defining your computational graph specific to machine learning right. So so this is not new and and super cutting-edge in terms of the higher level. But it's what is new is being able to abstract it into machine. You know abstracting infrastructure, into machine learning and and then is introducing get-ups, so we're going to dive into this example.

A

Has anyone here heard of this new platform called PI torch, hub or tensorflow hub? Let's see a show of hands, peiter job tensorflow, hub, okay, cool. So basically, what that is is a repository where researchers and engineers they can actually submit their trained models. Things like Bert G, GP to things like vgg, all of those neural networks and models that were trained on massive scale data or just models that you've trained can be actually just put in github and made available.

A

So you have that way of versioning that we discussed previously with your code and model and configuration right. So what we do is we're able to provide an integration, and this is all open source by the way, this whole open source, we're gonna, show. All of the examples are available online. We're able to have a direct integration, get-ups integration with the PI torch, our github, and what get-ups is. It basically means you keep in sync, your your github definition and your production environment.

A

So some, if something changes and github something changes in your production right and that is sort of your layer of sort of quality that you make sure that there's an approval process around that, and in this case we provide that get-ups layer between pi towards whenever a new change comes in then the the seldom gets triggered it wraps and container ices the model, and then the model is put into production. It could be like you know, a simple model. It could be. You know an a/b test.

A

It could be a a stacked set of models, etc, etc, and then we're gonna dive into that example. So so, basically you know. The first example was here: you can actually build your NLP pipeline. You can try it out yourself, but now we're gonna dive into into the next example, which is you know, building production level, piped or chopped models in kubernetes with ten lines of code, for it. To look like that, the ultimate objective is to separate concerns right. Your data scientist is able to use the models your Sree develops.

A

Person is able to manage them and monitor them right. That does the main thing is how to abstract and distribute concerns and separate concern, so let's actually zoom in so the process that the steps that you would do is you would basically first just create a wrapper in this case, for Python of your model create a wrapper of your model that has a predict function. So anything that has that you know takes an input and returns. An output can be wrapped.

A

In this case we can take nan PI race, but we choose to take the highly optimized tensor through through the byte serialization right. So what this does is a wrapper that takes some tensor, runs it through the loaded model and returns. The inference result right. It just so happens that when it's initialized it calls torch hub load and it loads the specific model right.

A

So what this basically does is it's a wrapper for tent for tighter job that is able to load models dynamically from PI to chop and once it's in production, you know when you send any requests, it's going to basically be able to, you know, run run everything through that model. So what we will do is we will just save this in a file so we're doing right, file in deployment image, PI, torch cell and deployment of Pi. Then we specify the requirements of that.

A

You know code of that Python code. This requires PI, torch, numpy, the image pill, library, and then we create a configuration library that says: what's the name of the model that we're sorry', the name of our wrapping class wrapper class, and then we just use the s2i tool to be able to build it.

A

These basically contain arises it and builds a docker container that has a REST API and a G RPC API, together with Prometheus metrics containerized, with with that wrapper. So now, if we spin up that that that container, if we pass the model that we want, it would basically dynamically pull that model from tighter job right. So we basically created our Selden configuration and every time that we trigger it, we just say I want imagenet or Inception b3 and it just basically pulls it and deploys it into kubernetes right.

A

So in this case, we define our like a Yama file that specifies that image that we just built so is that containerized pi-thirds wrapper, and the only difference is that here we have a replacer for the parameters that we pass. So this is our own Selden definition right, so you define the containers and then you define the computational graph right in this case. It's just a computational graph of one node and we're passing the parameter model name which in this case would be.

A

You know vgg, 11 or BG 19, or it can be Bert right, whatever it is, and what then it does. It basically deploys it. So here we're just actually doing a set replace so we're deploying the mobile net and then, when you actually run it, you can see that it's actually running, because we pass those parameters. It's called PI to shop and mobile net right m-net, and we can actually, you know, get the logs to see that it actually downloads it and then runs runs the application. And then we can just send rest requests right.

A

So we create a seldom rest client, but you can also do curl or postman or whatever, and we load an image. So this is the image of this dog.

A

We convert that image into tensors and we send the bytes of those tensors for getting like that request and response right, so we Pikul and then we just send the predict and then similarly here we're doing so here, yeah we're on pickling the response right, so is it sending the request and then we get the response, so it's basically a REST API, a G RPC and rest API that you deployed and now you can interact with write.

A

So ultimately you build your model in Python you build a wrapper and then you're able to just deploy it as many times as you want. You can have your replicas scales, etc, etc. You get all the benefits that kubernetes gives you and again. You know this example is online, so you can actually jump in and try it out. So that is for that. So again, you know check it out.

A

That's gonna be on this and again for the people that I just arrived, is github.com, slash, ethical ml and then you'll see state of machine learning, 2019 that you'll find that link and then other libraries to watch, because you know this is a whole ecosystem. You'll find this in the in the list. M leap serving focuses on optimized.

A

Serving of models, this allows you to use a serialized definitions like PI and py FML, as well as several other standards Oh in an X that you can actually use for for the deployment of your models. Deep detect is a unified API that abstracts. You know some of the most common libraries, and these are different sort of levels of abstraction. Right, like this one's focused primarily on unified libraries, so they don't really container eyes. So it really depends on the use case.

A

You know we try to focus on a more higher level, generic use case and now we're gonna jump on the third part on explain ability. So this is basically tackling the issue of black box model situations, and the reason why this is important is because in highly related sectors, if you are able to are predicting someone's approval of a loan, whether someone's loans should be approved or rejected, you know if, if there's a regulation in place like you need to be able to explain, a prediction ended up being in a certain way.

A

You know for various reasons, perhaps where there is discrimination of a specific group, whether it is a bias inherent or you know, unconscious, conscious or unconscious. You know whatever is. That is the underlying reason. Explain ability is a very important piece in to machine learning and you know explain ability needs to go beyond the algorithms. You know it is important to find the tools, but it's also more important to identify the process, and you know the process that you know we currently follow.

A

It extends the traditional data science workflow by introducing three steps, so data analysis, model, evaluation and production monitoring, and you know for the for the data assessment. The first part, you know it's identifying things like class imbalances, protected features, things like correlations data represent ability. You know all those things are very important to I be able to identify, but also to match with the specific domain for the model assessment and we're gonna dive into that is feature importance. You know local versus global methods, black box versus white box model, explanation methods, model matrix analysis.

A

You know in this case that is a feature importance. Permutation feature importance approach and then the last one is production. Monitoring is the question of what are the metrics that you need to put in place in order for your production environment to have a reasonable level of monitoring and alerts? Even perhaps right, if there's a mole divergence. What does that even mean because, of course accuracy?

A

It is context specific right, like you can't just arrive, and you know ninety percent accuracy in one use case is different to another, and the reason why this is important is because, when now going into production deployment of machine learning, you now encounter things like essays that you need to abide by right with microservices. It's easy right, uptime, you know you may have sometimes support, etc, etc, but with machine learning systems, there may be some metrics that you know need to be defined in place for the for that to reflect what value is being brought.

A

This could be things like accuracy. This could be things like precision, recall, etc, etc. A level of explain ability. You know what what would that even mean, etc, etc, and what we're gonna be diving into is something a very interesting piece is going on: the only higher level part of this leveraging, kubernetes and and more specifically, as I mentioned previously, the CR DS within kubernetes. You know leveraging this has allowed us to provide a higher level of abstraction for design patterns in machine learning.

A

In this example, specifically, we have a design pattern which we, which I call the explainer pattern in this case you're able to deploy a machine learning model which could be a black box model.

A

You know a containerized model, as we just showed that this could be like a PI touch more like, like birds or vgg or whatever it is, and then you would deploy a sort of like a respective expects explainer right, and this explainer would be able to explain why the model is doing the predictions it's doing right so once you would have a model doing predictions, you would have another model explaining that models, predictions, and that sounds really meta right.

A

That sounds like inception, but but it actually is it's more practical than you think and we're going to be diving into an example. For this last part, we're going to be using this library that we open source code alibi. So this is basically a model blackbox model, explanation, library and we're going to be using specifically this concept of anchor model explanations. So this basically ask the question: if the model made a prediction: what what pieces of that input?

A

What were the features that made that prediction that label in this case, you know- is the question of what made this picture a cat and it identifies what are the pixels that are how most predictive power for that specific prediction right. So this this allows for explanations of the model itself and we're going to be using that in our in our right. So in this example, we're going to be basically first building a income prediction, classifier right, so we're going to run first, are all of our components with helm.

A

Then we're going to train a model using the income classifier data set. So this is a tabular data set to be able to predict whether somebody earns more or less than 50k. We will have then a model that is basically trained on that, so we're using just basically a uh you know: a label encoder to be able to like process the data, we're gonna, create our preprocessor and then we're gonna create with a random forest classifier the predictor right. So we basically just have a model right.

A

We have a model, we're gonna treat that as a black box model, it doesn't matter. If it's a neural network or random forests, we don't care right, it's a black box model. When we run a friction, so we can see that you know the performance is gonna predict either either. You know a binary was a positive or negative class, and now we're gonna continue eyes and deploy the model. So again we create a wrapper.

A

So this wrapper, basically just you know loads- that pickup model that we just created and whenever you call this function, you know it takes whatever the input is, runs it through the model and then returns it. So this is gonna, be our API right, so once you create the wrapper, we basically define the dependencies again and build a container by just pointing to that wrapper, and now we have a containerized model.

A

We deploy it to cover Nettie's with the same definition, you know, with with cube codes, will apply so we're just saying like that container we just built. You know we want to rest API, FOA and then once it's in production we can actually just send curl requests. So here we're sending the data for that income classifier. So these are the features, and then you know we get the prediction here, whether it's positive or negative.

A

So here it was, you know, a negative prediction right so with what we now do, is we actually take one of the explainers, the anchor explainers, and we basically take that explainer and use that function of the model. So we basically give the model to be able to say well what of those? What from that model gave that prediction- and in this case it says well, it was marital status being separated sans sex female. What made that prediction being rejected right?

A

So that's what it basically does, but the cool thing is that we can now connect through you know that same you know. We create a function that reaches to that model. We deployed right and we now can actually make that explainer point to the production model right, so we're we're basically now, instead of explaining our local model, we're explaining that model in kubernetes right, so we're basically just saying well use this predict.

A

This remote function predict remote function, which is just sending the request there, and we can actually explain that that that remote predictor so now that we can actually explain the remote predictor, we can deploy our explainer right and we follow the same steps. We are going to containerize it by creating a wrapper right, so we just create a wrapper.

A

We containerize it with our script and then we push it to production, so we container I sit and we push it to production, and now we basically have, as I explained a model to be able to run predictions and then a model to be able to explain the predictions that it ran and we actually have a front-end that we can use to actually show you how those predictions would look like. So here we have a the income classifier. We can see all the all the logs.

A

So all of the predictions that have gone through that model and we can actually request well I mean if internets on our side, we can actually request the explanation for that specific prediction right so here we can see why it predicted what it predicted. So it's the same thing. We saw what shiny right so right now it allows you to just see what what we saw so now, it's cool if I wasn't cool, but you know now you can see what what what power it has so just to wrap up.

A

You know because this is the last few slides, they're all very cool for air frameworks, like le5 explained like I'm five shop, which I would recommend check out and xai which focuses more on that data. So with that said, you know we covered a high level overview of the ecosystem. There's still so much more to cover. Please go check out the ecosystem list. This is one of the most extensive lists of open source tools on production. Machine learning please contribute because we want to have you know a more and better understanding.

A

We've just added this week, a new section on adversarial robustness, which is really cool and we're gonna actually add that to the production set of tool set. So with that said, thank you very much. You know you can find the slides online. If you have any questions, I'm gonna stick around. So thank you very much for your time.