Continuous Delivery Foundation cdCon 2020, 20 Oct 2020

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: A CI/CD Framework for Production Machine Learning at Massive Scale (using Jenk... Alejandro Saucedo

Description

A CI/CD Framework for Production Machine Learning at Massive Scale (using Jenkins X and Seldon Core) - Alejandro Saucedo, Seldon

Managing production machine learning systems as internal data science infrastructure requirements grow, has uncovered new challenges which have required fundamentally different approaches to that of traditional CI/CD in software engineering. In this talk we will dive into the work we are doing at the SIG-MLOps and the CD Foundation towards developing the methodoloties that encompass best practices to continuously integrate and deploy machine learning in production at massive scale. In this talk we'll provide key insights on the core MLOps concepts, as well as a hands on coding example where we take a text classificaiton model through its training, deployment and promotion as canary and shadow deployments, which will also allow us to get deeper and more specific insight on our production environment.

A

Welcome everybody uh to my talk today on a practical, ci cd framework for machine learning at massive scale. uh My name is alejandro and a little bit about myself.

A

I am engineering director at solvant technologies, an open source uh company focused on productionization of machine learning models, I'm also the chief scientist at the institute for ethical ai, which is a uk based research center that focuses on the development of responsible frameworks for machine learning, the operation deployment and also um the acm council member at large, where we focus primarily um in my um contribution with the european technology policy committee as well as uh other uh communities, uh but today yeah we're going to be delving into uh why uh production machine learning is so hard and also how to provide a solution that scales across thousands of models.

A

So, as you know, when it comes to machine learning, there is a broad set of challenges that we face. This includes specialized hardware, uh complex uh computational crafts, uh compliance requirements of the uh assets that are deployed, as well as the reproducibility of components, and you know if you're interested in the in the space.

A

um I have previous content that you can check out as well, um but also today, what we're going to be asking is: how do you not only address production machine learning, but how do you deal with thousands of models across various environments with different versions of models that are being deployed?

A

So some of the challenges that you face at scale includes things like uh complex accountability. Structures of you know who should be providing sign of where advanced, observability and monitoring challenges when it comes to complex, specific machine learning, use cases, you know, governance and policies that need to be enforced uh not only once, but also throughout the life cycle of models. uh The shifting capabilities. uh Well, the shifting requirements on privacy as technology evolves, data sets uh can provide new insights and hence risk can be introduced.

A

Risk can be introduced, uh explainability compliance, uh another things like adversary, security, uh privacy, preserving machine learning between between others and when it comes to this uh perspective, there is also the end to end approach, so it's not only about tackling it from a bottom-up approach, uh but also ensuring that the principles and guidelines that are on the overarching perspective have the underlying industry, standards and regulatory frameworks.

A

And then, from that perspective you have the right tools and frameworks and platforms that allow you to really address that end-to-end, because, ultimately, you know we can have multiple discussions on how uh principles uh around algorithmic bias are important. But if you don't have the underlying infrastructure to enforce them or to ensure that they're being uh um you know tracked, then you know there's no point on on those higher level discussions.

A

So for that perspective, at the institute, we came up with a set of principles for specifically practitioners of uh ai, not just technologists but broader teams, uh things that you may find you know quite reasonable, like human augmentation bias, evaluation, etc, etc, which we have provided for some practitioners to to adopt right. But, as we mentioned, principles are not enough right.

A

It's necessary to go level deeper, and from this perspective, there is a set of um um initiatives that are aiming to tackle this challenge of uh what is now referred to as the concept of emelops and what mlops is defined, as is as the extension of devops to machine learning as a first class citizen and more specifically, is the infrastructure and tooling that is used to productionize machine learning and then scale this this.

A

This production machine learning capabilities, and there are several working groups that are currently exploring this further one of them um is the sig envelops committee, which uh you know us and many other open source uh working groups. You know it.

A

It is led for practitioners by practitioners and, as you can see, envelopes in itself is the best practice across multiple disciplines that are not just from the technical aspect, but have to go beyond the algorithms, ensuring that they engage a broad range of other policy experts or domain experts which are still, you know necessary for this and the key requirements to address here. It's going to be the principles of orchestration.

A

How do you orchestrate thousands of heterogeneous models, advanced ml insights, how you ensure that models are being monitored not just through the usual metrics, but also through advanced metrics, like explainability, outlier detection concept, drift, etc and then lineage? How do you make sure that, throughout the end-to-end life cycle of a machine learning model from the time that the data is trained, sorry, the data is cleaned uh processed? The model is trained, the model is deployed, the model is promoted.

A

How do you ensure that there's, lineage and and auditability across all of these different phases of of your machine learning, and specifically the first question you know from the study principles that we're going to be addressing is: how do you go from you know, mortal artifacts or model code into fully fledged deployable microservice that is providing logs tracing, metrics, etc, and for for starters, we're going to be using this library called seldom core so seldom core.

A

It allows you to go, as you know already alluded to from python code or java code or r code into a fully fledged uh microservice right, and the way that this is going to be possible is by uh you know, by being able to leverage not only a simple deployment but being able to deploy a simple model with you know, a rest or jrpc api being able to deploy an a b test with multiple different models that can be actually, uh you know tested against whether it's performance you have more sophisticated.

A

uh You know deployment strategies like multi-arm bandits or shadows or canaries. We can have uh things like these joint feature transformers that can be reusable across different pipelines. uh You can have other components like outlier detectors and as we're going to cover later on in the in the point about ml insights, things like explanations, and we actually have a broad set of um open source examples that you can try out um that you know.

A

Ultimately, we encourage- but you know specifically in this context, we're going to be delving into the more holistic question of not just productionizing one model, but thousands of this of these models and of course, when it comes to the deployment and containerization of a model, you have several principles in play. The first one is the auditability and provenance making sure that your uh uh components are completely um consistent right.

A

So you have your data that was used to train the model, the code specifically for the model and then the input and output data that goes through the model when, since once it's deployed- and this is quite important- given that you want to introduce a level of reproducibility to your stack, then you also have the the the requirements that don't go just on a single model, but across your entire complex uh data flow pipeline right.

A

So if you have multiple different models, you want to make sure that there's also the ability to reproduce the way that, though, that the data circulates across to ensure that you can replicate a specific data point that was processed for debugging or editability perspectives.

A

And then you have the metrics monitoring to ensure that you have the ability to monitor the models that you deploy, so whether it is for for for load testing to ensure that there are, you know no outliers in the performance, but also, as we will see later on, for actual machine learning monitoring um and then, of course, the other trails of everything that goes in and out of the model, making sure that you're able to understand and keep track of uh you know the relevant uh processed uh components.

A

Specifically, you know whether it is for audit purposes or if you want to then try to debug something that happened. You know at another point, but then there's the question also of how do you not only deploy a single model, but how? How do you ensure that you're able to continuously deploy multiple versions of heterogeneous models coming from a broad range of different model sources right? So this is not just your specific python artifact store, but it may also be your ml mlf store your spark ml store your r studio models.

A

How do you ensure that there is a continuous integration in continuous delivery, and for this we introduce some core concepts that will be abstracted beyond the framework but are going to be using this uh framework called jenkins x, so jenkins x is basically a project that was created uh from you know, as you may recall, from the name uh the jenkins team. However, it has no relevance in its underlying code beyond just the name and the team of course.

A

However, this project leverages the cloud-native components of kubernetes to provide serverless capabilities on continuous integration and continuous delivery, and this will become a little bit more uh intuitive as we go as we go ahead. So the way that we would look at at this perspective is going to be uh in two parts: the continuous integration and the continuous delivery first on the cips.

A

Let's assume that we have a a you know, data scientist and we're going to see how it interacts with with jenkins, as soon as a new model is created that will be able to go into the model, implementation, github repository or version control system repository, then, once we want to deploy it, we would do this through the concept of githubs and github, says deployment uh as code through version control so similar to how you have uh other. uh You know things like infrastructure as code deployment.

A

This code is basically keeping all of the deployment uh state in a version control system so that it reflects the current state of your cluster and you're able to to revert back. So what that means is that whenever you submit a change, you're able to interact with jenkins x, not directly, you know by sending commands, but through the pr workflow uh what through what they refer to as chat ops right.

A

So this is basically through the pr comments, you're able to send commands to jenkins x, which is in your kubernetes cluster, and these commands are received uh by the brow system. You can put comments such as slash assign, which could be like a github user.

A

This could be like slash, retest or slash test, or if you actually want to approve a apr, you can put just slash approve and what you have is other commands like hold life cycle, or you know, perhaps more useful commands like slash joke that actually just posts, a joke on the on the pr um and then similarly the the jenkins sex workers. What they would do is they would trigger the actions that are respected to the commands, and this would be using the serverless architecture which in this case, is using tekton.

A

uh It's a frame, the underlying framework that runs the the ci steps, and in this case you would have your build commands being carried out by the underlying you know, scalable framework, which in this case you can see the intuition is all using the the the traditional um um version, control uh workflows right, submitting a pr ensuring that people review it and then the right individuals are able to approve or reject the pr and then once you're able to once. You want to promote a specific deployment. You can do so similarly through the pr workflows.

A

So here you can see the multiple different version, control repositories that are into play, and here you can see some of the examples. Once you open a pr, you can actually interact with uh the jenkins backend through the the the githubs commands right, slash, approve or slash joke or slash roof. uh You know all of this all of these comments, um and you can see that with this we have been able to uh uh help uh us as seldom core skills in its contribution. So seldom core is an open source framework.

A

We have over 2 million installations and, as you can imagine, the contributions have also grown from our open source user base.

A

Having this, uh you know, cloud-native capabilities has definitely helped us scale, and this is on the continuous integration piece making sure that tests are wrong, that you have all of your pr's, you know being able to scale as the contributions come in. However, you know, as I mentioned, a lot of this is led by the concept of githubs, so this is basically ensuring that the actual deployment is all happening through a version control system.

A

So once you go into continuous delivery, actually deploying data into production or or deploying an application into production, this all happens through these version control, githubs, workflows and specifically into this. What you would see is that if you want to deploy a model, let's take this specific uh uh deployment structure. Let's say we have a production namespace with these models there a staging namespace, and then you know in the specific jenkins environment. That's where jenkins is running once we actually uh create or train a new model.

A

Let's assume that we want to actually uh roll out a new version of our models. What we would do is as soon as the the model is ready. It would actually go through the continuous integration steps and once you know we merged those uh changes, then the actual, continuous delivery would be triggered. What would happen is the actual um configuration as a helm chart will be published right.

A

So this is what we're gonna then be able to uh use for promotions across different environments, and then we can, actually, uh you know, leave everything else for jenkins x. To continue so jenkins x would actually open up a vr into the github's environment. As soon as the the github's environment changes, then you can see that the actual models get you know updated in this case we've upgraded into version 1.1 and 3.1, and whenever we're ready to actually perform a promotion, this is also performed using githubs.

A

So automatically jenkins x is able to open a pr which says I want to promote uh these specific models into this environment and what once it's approved, uh then that's when the actual uh changes would be reflected into the environment, so the configuration would all be stored, and this allows you for more complex and sophisticated deployment strategies like, in this case, canaries or shadows.

A

So you know here we have a percentage of the traffic being, uh you know streamlined, and this allows you to actually uh deploy and uh scale your continuous delivery at you know broader sort of scales, primarily through these gitops components.

A

As then you're able to have the same uh standardized uh interface for the flows of each of the models that go through the different environments, and we actually have a hands-on implementation that you can check out. uh You know we encourage you to try it, and you know we also have not only the one with genuine sex, but also the one with jenkins. um You know if you prefer to actually run it in a more sort of old school.

A

uh uh Well, I say old school, but you know that's what kind of happens today as of today right and then the last thing is well. So now we have a means to deploy models, but how do we now achieve the same perspective for uh other components like things that allows us to explain models or to identify outliers of the models or to provide concept, drift, detection on the models and specifically into this? We we do need to be able to identify the ways in which this can be delivered.

A

um You know similar to how we're deploying and productionizing our models and the way that we're able to do this is through this framework called alibi.

A

So alibi is basically a set of state-of-the-art algorithms for explainability outlier detection, other cell detection, which in turn allow you to not only provide you know state-of-the-art tools themselves, but also be able to introduce processes that involve domain experts beyond technologists.

A

That would be in charge of performing assessments of data assessment of model, uh an assessment of production model performance right, and this is key- that it's not just about tools, but also about process and the humans throughout the different touch points, and what this allows you to do is to leverage uh what would be uh inference. Data assessment, for you know, class imbalances particular feature assessment correlations.

A

You know model assessment, for you know all the other things like model, specific methods. You know abstraction of domain knowledge and the features uh uh future importance, etc on a global level and on on a local level ie for data points. What are the actual features that uh are more um influencing the the prediction of the model and then also, of course, the production monitoring right. Once you deploy a model, can you still perform explainability of data that goes through it?

A

Can you deploy uh some outlier detection components and the key thing here is well: if we have thousands of models, we will require to introduce this uh advanced monitoring perspectives, but we need to abstract these capabilities into a means that allow us to really provide this, that massive skill right and for this uh you know not only from the explainability perspective, but also into the security uh perspective. You know, there's a concept of adversarial robustness.

A

uh You know security that that that models could be exploited on such as by providing you know, perturbations on the probabilistic inputs that could trick the models right. So how do you actually abstract these core concepts into means that can actually scale in your infrastructure?

A

And this is something that we have been able to uh introduce through a similar means uh to which southern core provides you with the deployment of models uh we we also provide with the ability to abstract explanations or explainers into a similar means. So, whilst your model provides predictions for every input data, an explainer when you send a request, it provides explanations by reverse engineering, the model.

A

So this is basically an explainer pattern that, even though the underlying explainer inside of the model can be very specialized, this as an infrastructural component is something that we can still manage as a as an infrastructural pattern and as an I.t component. That then are our devops infrastructure and those continuous integration workflows can still allow us to to keep the full provenance across right and then, similarly for outlier, detectors and adversarial detectors, we can also abstract them into infrastructural patterns, and you know if you're familiar with with cloud native, uh uh you know approaches.

A

This would be through actual uh custom resource definitions that that provide you with that level of abstraction on these components that can be very highly specialized, and so with this you know this is really important, given that what we have done so far is take very complex, uh specialized machine learning, concepts like models like explainers, like adversarial detectors and then take traditional, uh um um get a ci cd and a best practices of of of software workflow to be able to provide frameworks and and and workflows that scale to hundreds or thousands of models, and not only models, but also the the components that surround those models themselves, such as explainers adversarial detectors, and the reason why this is important is because this provides us with a with a with a ways to standardize and scale what are often highly complex machine learning concepts and then similarly providing full provenance on uh the model history uh through githubs right.

A

So we can actually revert back uh these components reproducibility through simplified rollouts. um You know operational process with allocated touch points that you know, go beyond the technical stakeholders and then also go governance and eligibility to reduce the risks at scale.

A

You know when you have deployments of hundreds or thousands of models, so this is important, because this is also a challenge when it comes to just microservices right, and this is why adopting these cloud native um best practices and just bringing them into into domain uh uh specific areas like machine learning is what allows us to really extrapolate this.

A

These best practices that that really allow us to scale as as broader technical and broader teams- and um you know, if you're curious, on what other of the tools available, we actually create a large list of production, open source tools which you can actually visit in. You know this github repo and, if there's any missing, please feel free to to add uh to that so uh also massive shout out to xkcd.

A

If you notice there's a lot of uh artwork, uh you know throughout the presentation um you know the work is awesome, so you know like always uh check out and support their their awesome work, uh and with that said, uh thanks everybody for uh your time. um I hope you enjoyed this presentation uh on a practical guide, a practical, ci, cd framework for machine learning at massive scale. So with that said, uh you know I'm more than happy to to take questions uh and, if not uh you know, the the examples are on the repositories.

A

uh Outline so feel free to try them out or to add suggestions as github issues so yeah. Thank you very much and uh yeah. I hope you enjoy the rest of the conference. Hello, uh yeah! Thank you guys. Thank you, everybody for joining, um I mean, I guess I assume this is working, um but yeah, I'm more than happy to cover in a little bit more detail.

A

uh I was currently answering some of the questions on the chat uh providing some of the links. So in case you wanted to try out some of this things that we covered. um They are in the documentation and there's basically jupiter notebooks that you can test both in uh jenkins as well as jenkins, x um and, of course, the uh approach that we took uh for this is, as with anything in software, uh one of the many ways to tackle it right so now.

A

This is not going to be the ultimate um correct solution and that's why, uh if you do try it and you have any suggestions, please do feel free to share them whether it's on the on the slack group. You know, I share the link for where you will find the slide group as well, as you know, opening an issue on the repo um which I I also shared um so so yeah, so so please feel free. So I think there's there's a question.

A

uh I think we have uh five minutes, so we could still cover a few questions, so it says most of what you showed is for deploying models into production using githubs. Do you also work on a ci cd for training and improving models over time, I.e, uh offline versus online?

A

um So that is a very good question um specifically in regards to the answer: there are two parts to it. The first one is: um is there an option without githubs and the second is: is there a possibility of adding uh the concept of offline online retraining of models?

A

So the answer for both is yes and, as you may already have assumed the example with jenkins classic not jenkins x doesn't use githubs, uh not because it can't use githubs because it definitely can you can. You just have to actually push to the to a repo and then sync it, but in this specific example, it just uses the traditional pipelines that uh pushes that um so now, uh in regards to the second part of, can you have ci cd pipelines that also take into account the retraining of models?

A

So this is actually quite an interesting point. The answer is yes, and we actually have seen a lot of production use. Cases where either automated automatic uh pipelines are created, created based on rules or uh manual pipelines are created, based on triggers that allow for the steps in the pipeline to actually run the training of models right.

A

When would you use automated pipelines when you actually have uh you know semi self-labeling data sets in production, which then you can actually retrieve and assuming that you have a reasonable distribution of the data, and you have your own policies to determine that you'd be able to automatically retrain the models.

A

Otherwise, when it comes to the offline, we have seen a lot of uh use cases where now there is an an an into sort of an ambiguity or a combination between the world of etl, the extract, transform load and the world of ci cd, where you tend to now do some of the steps in the etl space, such as training a model now in the ci pipeline. So so so the answer is yes, and I think um that's something that we have also seen.

A

uh That has been quite interesting uh and then another follow-up question is: can we get a link for the jenkins uh classic example? So the answer is yes, so the link that I sent above actually contains uh both. So if you scroll down, you will be able to see the link to the ci cd jenkins classic and the link to the ci cd with jenkins x. So more specifically, that's the link to jenkins classic- and this is the this- is the link uh to drinking sex.

A

um But yeah, I mean, as you also know, the the one with drinking sex uh you know was created. You know uh a a few months ago now, like maybe you know seven eight months ago uh and as you may have seen, how fast jenkins x moves. uh If you do try it, um there will be some uh changes that will need to happen.

A

So we're probably gonna have to update uh the example to use the latest uh thinking sex, uh but at the same time, the principles and the workflows, which is the most important part, are still consistent right.

A

The ability to understand how a model would be able to not only transition into a specific environment, but also how you would be able to promote it into other namespaces and, more importantly, how you can actually use githubs to do the promotions, and I think actually um one thing that may be an interesting point to also mention which you know at this point is more just rambling, but it's still interesting is that the the one of the things that we have identified is that approaching continuous integration and delivery of models using um using githubs also enables our environments to be extensible to things like multi-cluster strategies and the reason.

A

Why is because, when you, when you use githubs the way that you're able to deploy a model uh into the cluster, you would have to just push it uh into the um into the version control repo, as opposed to modifying the cluster right. So what that means is that if you have another cluster, the only thing that you need to do is to connect that repo to the other cluster.

A

So then, you can have very complex deployment strategies of models that you can then promote across completely different environments uh using using githubs, and I think I think that that, for us has been quite useful uh when uh engaging with uh very complex.

A

uh You know deployments that that have to move from production from staging to production uh involving actually different clusters right, so that has been has been quite an interesting point that we've seen um but yeah so so um happy to to answer any other questions um and if not uh also I'm sure, there's gonna be tons of other sessions.

A

uh Tons of other exciting sessions uh to still cover um so yeah, maybe I'll give it a couple of minutes uh in case some. uh Anyone else uh has further questions um and uh yeah I mean I can also answer them uh offline, so yeah I'll give it a couple of uh of minutes, or or at least sometime um and yeah, and thank you again uh for everybody uh for joining uh yeah. I hope you enjoy it. The video is gonna be online as well um yeah.

A

So I'll I'll wait a couple of minutes to see. If there's any questions, and if not, we can, we can wrap up cool. I think jennifer just mentioned uh uh next session starts in five minutes. So if you're planning to attend uh yeah make sure you click on the sessions for the next presentation uh and yeah, I mean I'm still open. If anyone has has questions um on this.

A

And venugopal yeah thank you and if you want uh to test uh more examples now, there's there's a there's, a broad range of um jupiter notebooks that you can try um but yeah, um both in the southern side and in the jenkins x side. I think there's, there's always enough fun awesome! Well, I guess we can wrap it up there and uh yeah. Thank you again to everybody for joining and I'll see you around in the conference.