Cloud Native Computing Foundation Kubernetes Community Days (KCD) Chennai 2022, 5 Aug 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: MLOps with Flyte by Samhita Alla

Description

As the discipline of machine learning operations (MLOps) matures, it’s becoming clear that, in practice, building and deploying ML models pose additional challenges compared to the traditional software development lifecycle. The development and deployment of ML models require certain features such as reproducibility, recoverability, maintainability, audibility, scalability, and compute management, among others. Using Flyte, a data- and machine-learning-aware open source orchestration tool, we’ll see how to address these challenges and abstract them out to provide a broader understanding of how to surmount them.

In this talk, Samhita Alla:
● Presents the challenges in the context of ML model development and deployment
● Introduces Flyte and dive into the ways through which it provides solutions
● Elaborates on the ML-specific features offered by Flyte
● Does a best practices walk-through to develop models fast and productionize ML code effectively

A

Hello: everyone, my name is samhita, I'm a software engineer and technical evangelist at union ai, where we are trying to make the orchestration of machine learning easy to use. I am here today to talk about ml ops with flight, remove the barriers to successfully implement machine learning for production workloads.

A

The crux of the stock is to know how machine learning can be productionized using a kubernetes native platform which, in our case is flight.

A

Let's get started by understanding how ml ops differs from devops to automate and smoothen the process of software engineering. We inculcate and abide by a set of practices which we call as devops devops or development operations refers to bringing together the development, testing and operational aspects of software development.

A

The best practices include continuous integration and delivery infrastructure, as code monitoring and logging communication and collaboration and microservices.

A

Dev needs devops; likewise, ml needs ml, ops, however, ml ops is a superset of devops. This is because models could deteriorate over time. Why is that so? Models and data are mutable, they require constant experimentation and iteration data could be very complex, it can be huge and it may need to be pulled from multiple data sources, infrastructure overhead. We may have to take care we'll have to take care of the latency, higher processing power, compute requirements and next comes hardware requirements.

A

More often than not ml code needs gpus, so we'll have to set up gpus, take care of the memory, requirements etc, and, lastly, we'll also have to take care of the scale of operations, meaning the scalability of machine learning models we'll have to ensure that, in addition to the previously mentioned, challenges we'll also have to think about arm which stands for auditability, reproducibility, recoverability and maintainability auditability is the ability to backtrack to the error source when a pipeline execution fails.

A

It also includes data lineage, which means to understand data, that's being used and transformed over time. Reproducibility is the determinism of a run. Are we able to reproduce the run given the same set of inputs?

A

Recoverability is the ability to recover when there is a pipeline failure and maintainability is the iterative development of machine learning pipelines and enabling team collaboration.

A

Ml ops comes with technical debt. If you look at this diagram, ml code is only a tiny fraction of the real-world ml systems.

A

We'll have to take care of a lot of components in ml, ops, say, for example, we'll have to take care of resources we'll have to take care of serving we'll have to take care of monitoring, etc, and hence we need a cohesive platform that brings together different components, teams and models onto a single platform, components, meaning the components that I've talked about earlier say: surveying monitoring teams, because uh in mo we can have multiple teams pertaining to multiple domains, and the teams may want to work together.

A

They may want to collaborate, and hence the platform has to support team collaboration multi-tenancy to be specific and then models uh we usually in machine learning. We usually do not work on a single model. We need multiple models. We work on multiple models, so yeah, that's how that has to be supported too.

A

Moreover, we also need an amalops platform that is scalable, extensible and reliable. Meet flight flight is a kubernetes native workflow automation, platform for business, critical machine learning and data processes. At scale so flights built on top of kubernetes and it's a workflow automation platform, that's that can be used for mission, critical machine learning and data processes.

A

The goal of flight is to provide a reproducible, incremental iterative and extensible workflow automation platform as a service for any organization. Its focus is on user experience, reliability and correctness. If you look at this diagram, we are separating machine learning. We are segregating machine learning and operations. So, on the ml side we have user teams and on the up side we have flight and platform team user teams usually consist of data scientists, ml engineers and so on, and they interact with the machine learning frameworks and on the platform side.

A

On the op side, we have platform engineers devops engineers who deal with cloud cluster management, resource management, so flight provides the capability to separate uh ml and ops to separate their concerns to separate their responsibilities um and yeah. That's that's a very important requirement of mo ops.

A

The basic entities of flight are workflow and task. Workflow is a sequence of processes through which a work through which a piece of work passes from initiation to completion and task is a fully independent unit of execution and a first class entity of flight.

A

So simply speaking, workflow is a collection of tasks that are arranged in a specific dependency graph. So if you see this, uh if you see the diagram task, two is dependent on task. One and task three is dependent on task two, which in turn is dependent on task one. So that's how we model a workflow and a task.

A

Let's look at the challenges of ml orchestration, that flight solves, develop incrementally and constantly iterate at scale. So let's look at the diagram, the steps in the diagram to understand what this means. The first step is to start with a job, be it a spark job, a training, job or a future engineering job. The second step is to scale the job. Third step is to create a pipeline and test it locally. Fourth, is to execute the pipeline on demand.

A

Fifth is to run the pipeline on a schedule or in response to an event and sixth, is to retrieve the results or the outputs. So this is a generalized pipeline architecture that flight supports, which in turn, helps to develop the code incrementally and also constantly iterate at scale and flight also supports having a centrally managed infrastructure which uh the platform teams can take care of. So there's a separation of concerns that flight supports, and hence uh we can uh we can.

A

We can only have one centrally managed infrastructure that the platform teams can focus on and the user teams can focus on the machine. Learning models and flight also supports access to resources like gpu, cpu and mem through code, so it's infrastructure as code, and it also supports automatic scaling, knobs to control cost, multi-tenancy uh architecture etc, and as for efficiency, it supports spot machines and model check pointing which we are gonna.

A

Look into later next comes in ml experiments require similar algorithms with different parameters or inputs, and hence there can be altering behavior based on inputs. So if you look at this diagram in the first, uh the first one, the first diagram- we have input as m1, and this is the graph we have got. This is the dependency graph we have got and for input m2. The dependency graph is the same as the dependency graph we have got for n1 and for input n. The dependency graph is totally different.

A

We have an additional node and also dependency structure is different, so this is a very common scenario and machine learning and in flight this can be addressed using the dynamic decorator.

A

Next comes memoize recover and reproduce. So, let's look at the diagrams again. We have three nodes here right. We have three nodes. The first two are successful and the third one and the third one we are basically storing the output of it in cache.

A

So when we run the code again when we run this pipeline again using the same input, what happens is uh the the first two nodes are successful, all right and for the third node flight uh fetches the data from cache instead of executing it instead of re-executing it that's the advantage with cache and next comes the recover feature of flight wherein we have a pipeline again right. The first two are successful and the third one uh failed. The third node failed.

A

So what happens is when we run this uh again when we run the score again with the same input. What happens? Is it recovers? It's not just running, but it's hitting on the recover button, that's available on the flight ui that basically runs it again. So the successful executions are copied and flight starts running from the failed executions, but it doesn't run the successful execution. So that's the advantage of using the recover feature in flight and there's also retries. If you see this, uh you can set retries in the task decorator.

A

So when you set retries what happens? Is it tries to run the pipeline again in the case? It fails.

A

Next comes collaboration and organizational scaling. So if you look at this diagram, we have two teams right: team: a and team b team, a has two pipelines which are using the which are using a critical or complex algorithm and team b has a pipeline which uses the critical or complex algorithm, that's available or that's being used, or that has been built by team a and there's also another pipeline wherein uh which uses a pipeline, that's built by team a so uh how uh flight implicitly provides support for uh reusability.

A

That's that's implicitly provided by flight and for the separation of concerns, there's a concept called project and domain in flight. What happens is uh project and domain basically mean that uh say: okay, let's understand that by an example uh through an example, so we have team a and team b right, so team, a per attention pertains to one project and team b pertains to a different project. Now, both team, a project and team pre project, have three domains: each right, one is development, there's uh staging and then there's production.

A

So that's the project and domain uh concept. So that's how we can separate the concerns of team, a and team b and, as mentioned previously reusability is implicit. You can just reuse the workflows from different projects and domains so that shouldn't be a problem.

A

Next comes extensibility of flight, so that's uh supported by flight. uh The extensibility is provided at three levels: at user level, at platform level and at organization level. At user level. uh You can, you know, use flight kit to add your plugins and integrations at platform level. If at all, you want to extend your platform in terms of distributed, training, support sparks streaming, that's possible through flight pack and plug-in, and you can also extend organizations say you may want to control costs or you may want to bring capabilities in house. That's possible too.

A

Next, let's look at the user level. Concepts of flight first comes tasks, so task is the building block of flight. It's the smallest unit of work in flight that basically accepts inputs and gives our gives outputs right and it has a strong type interface. All the inputs have to be typed and it can also map to a backend execution. Plugin. It's decorative version, language and framework, independent next comes workflows, so workflow is a collection of tasks that are arranged in a specific dependency graph. So a workflow is declarative.

A

It models, data flow through tasks. It has a strongly typed interface and yeah, it's declarative version and dynamically recursive. This is an example of a flight code snippet in python. We have two tasks and a workflow. This workflow is basically uh it basically encapsulates all the tasks. So here we are encapsulating, total spend and total spend is in turn dependent on pay multiplier.

A

So this is the dependency we are uh basically trying to tell and that's what flight captures uh we are also setting cash to true and cash version and the task for the pay, multi multiplier task, and when we modify the cash version, what happens is flight invalidates the previous cache and it tries to run the task again. So that's the advantage of using cache version. You can also run all of this code locally, just like calling a python function by sending the required arguments.

A

If at all, you want to scale your code say you want to specify resources for your tasks right. How do you do that? So in the task? You just specify your resources say you want to specify cpus a number of cpus memory. Gpus you can do that and if, at all you have spark code, uh you can specify the spark cluster configuration by installing the flight kit spark plug-in. This is product configuration and yeah. That's that's just uh that's the configuration for this block spark cluster and you can also specify retries.

A

So you are saying retries as two. What happens is if, at all the task fails, it tries to run it again right it if it's two, so it tries to run it twice. If at all it fails again, it runs it again. So it's twice and then yeah you can also create a launch plan. uh Launch plan is basically a plan that flight uses to launch or trigger a workflow, so calculate spend is the workflow we wanna associate this particular launch plan with.

A

We are also saying that this is the schedule, meaning run this every 10 minutes, and we are also saying that uh if at all, the workflow fails send a notification, email notification to the uh to a set of recipients so yeah next, let's look at workflow modalities, so dynamic is evaluated at runtime, whereas workflow is evaluated at compile time. Here's an example of uh yeah dynamic workflow.

A

We have a dynamic workflow and we have a fur loop within it where we are looping over multi-train, multi-valve multi-test, which are the parameters of this particular task which of this particular function. So uh the number of times this loop is going to run is dependent on the length of the list that the user inputs. So that's, that's the dynamism.

A

We uh that's the dynamism, we're uh I was talking about, so uh this dynamism uh is captured by flight at runtime and that's when the graph structure is built right and also a point to notice uh within the for loop. We are calling two tasks, one is fit and one is predict, so this dependency is evaluated at runtime awesome.

A

So how do you execute- and you know, execute the flight workflows and interact with the flight pattern? You can do that through flight ctl, which is the flights command line interface. You can fetch and create executions. You can fetch all the results of your executions. You can visualize your workflow through flightcle and you can do all kinds of stuff that you want to do with the flight back in.

A

And you can also use a programmatic python api to interact with the flight backend and that's called flight remote. You can use flight remote to you, know, trigger a workflow or fetch the results and do again do all kinds of stuff.

A

Let's look at a couple of pictures uh to understand how the flight ui looks like so here's a graph that cap that's capturing all the dependencies between multiple nodes, and this is a workflow page. So whenever you open a workflow and on the flight ui this this is a typical uh uh ui. You know, uh so we have multiple workflow versions. uh You can. You should be able to view all the workflow versions.

A

You should be able to view all the executions and you can also launch a workflow just by a click just by click of the button uh launch workflow button. So when you click on that a form opens- and you can give your inputs- you can also say, give a service account that you wanna associate your run with. You can also modify your workflow version, the version that you wanna use and so on- and this is the uh error.

A

This is how the error message looks like if at all, there's a failure, so this helps with debugging. You can debug with ease and uh flight also tells if it's a user error or system error shows you the trace pack and the error message.

A

Now, what are the features uh that are? What are the features of flight for platform? Folks? So if, at all there's a platform failure, you can recover instantly using the recover button, you can manage your customers transparently. You can set platform by defaults project domain resource limits, so you can use spot machines. You can set compute limits.

A

So all these are the features that are available for platform folks and now, let's look at the kubernetes side of flight, meaning the architecture.

A

So let's try to understand all the components that are here starting from the user, so the user authors workflows using the flight, sdk user, compiles them and then registers them using flight cli, which is flight ctl. In our case registers, flight ctl or flight remote and the registration happens after the registration happens. uh Basically, what happens is the code gets registered and they are now available on the flight admin? So that's where we have all the workflow and task information.

A

uh The user also builds a docker container to uh where, where where all the dependencies are present and after the registration happens, you say the user is the user wants to trigger the workflow. So what happens? Then? uh Flight propeller comes into the picture, so flight propeller is a kubernetes operator and it's part of data plane now. Flight admin is the control plane and flight.

A

Propeller is the data plane, so flight propeller executes the workflow guides the workflow to completion, and during that process it can interact with other kubernetes operators, or it can also interact with other kubernetes objects or some external services, and it guides the workflow to completion and stores the result in a cloud. Blob, storage and flight admin gets the results from the cloud blob storage and shows it on the flight dashboard which the user can view. So this is the work. This is the basic architecture of flight.

A

Next, let's look at the flights ml features. First comes intraodastic, pointing so running. Multiple epochs or different iterations, with the same data set, can take a long time. Hence the bootstrap time may be high and creating task boundaries can be expensive, so we have already looked at retries or recover mechanisms available in flight. Now, when you recover a workflow, what happens is it's recovers from the failed node, but not from within the node?

A

If at all, there's a failure within the task, it doesn't know that it doesn't know that it just runs the whole task or it just runs the whole node, but in machine learning we can have tasks wherein uh there can be multiple iterations within it. Multiple epos, it's compute, expensive time, expensive and we want to. We may want to capture the exact failure point and run from that point. So in such a case, we can use intratastic pointing flight offers a way to checkpoint progress within a task execution.

A

So how? How does that happen? How how do you do that? uh So this is a flight task right in that you can initialize a checkpoint, it's available in the flight kit, library and after you, you initialize the checkpoint.

A

You just have to write the required or the concerning you just have to write the data to it, and so what happens is this opens up the opportunity to use alternate compute systems with lower guarantees, like aws part instances, gcp preemptable instances which offer great performance at much lower price points as compared to their on-demand or reserved alternative? That's the advantage of using intra-tested, pointing especially when building machine learning models.

A

Let's also look at union emma, which is uh which is a an interface that's built on top of flight. uh If you look at this diagram, so we have union ml app right. uh If at all, you want, if you're at all, you want to deploy this union mo app, you can deploy it either on a web app or on the flight back in using flight kit and fast api, and there are train and project endpoints respectively. So union ml app provides it's like the easiest way to build and deploy machine learning micro services right.

A

It's it because it's flight compatible, you can easily deploy them and you need not really think about mlaps.

A

Let's, let's look at how the union ml code has to be written. You know the anatomy of a union ml app first import the required dependencies. We are also importing the union mo library and then initialize the data set and model here.

A

The data set that we are using as digits data set and the model is logistic regression, and then we have to initialize a reader method that basically reads the data or trainer method to train the data predictor to predict the data to generate predictions and then evaluate it to evaluate the model so pretty much it. You just have to write a couple of simple functions and then uh boom you or you have a uh you: have a production ready ml application with you after you deploy it.

A

So you'll also have to make sure that you decorate your functions right with the required decorators in order to keep it in sync in order to keep the code in sync with the flight back end because internally, these are tasks and workflows, and this is how you deploy right. Local. If you want to execute locally, it's pretty simple and if at all, you want to create a prediction: micro service, that's simple, too!

A

Here's an example using fast api, so you just have to trigger the server you just have to spin up the server, serve the model and then hit the end point to get the predictions.

A

Finally, an ecosystem of integrations is what we are planning for, so uh we currently have integrated union ml with fast api, but we have plans to integrate it with streamlit and flask. We are also planning to integrate it with other hyper parameter, optimization, libraries, machine learning, libraries, aws, lambda or serverless, and even driven service and so on.

A

Lastly, let's look at a couple of envelopes: best practices, yeah, so retrain models on a cadence. That's that's very important to keep your model updated version. Your data and code have a backup model for generating predictions in the case when the main model fails have ml related tests in ci sanity check the data, meaning ensure the quality of your data.

A

Monitor your machine learning models maintain a clean project structure, meaning segregate your model, and data code have pre-commit hooks to ensure the quality of your code.

A

Finally, with flight, you can code ship and scale ml pipelines with ease yeah. If at all, you want to learn more about flight here. Other references take a look at our website, documentation and repo. Thank you very much.