Cloud Native Computing Foundation Kubernetes AI Day EU 2021, 14 May 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Embrace DevOps Practices to ML Pipeline Development - Tommy Li & Yihong Wang, IBM

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Embrace DevOps Practices to ML Pipeline Development - Tommy Li & Yihong Wang, IBM

DevOps practices are broadly adopted to facilitate the software development, testing, delivery and deployment cycles. It also smooths the transition of a software application from development to production. Kubeflow is a machine learning (ML) toolkit on top of Kubernetes. Kubeflow Pipelines with Tekton is an end-to-end workflow orchestration tool which manages the pipelines execution on Tekton backend. Enabling DevOps practices to complete the integration of end-to-end workflow is part of our journey while developing kfp-tekton. I will share the experience about consolidating ML end-to-end scenarios, Build/Deployment status dashboard, Slack channel notification into the DevOps process alongside the Continuous Integration (CI) and Continuous Deployment (CD) on target Kubernetes cluster. I will also cover the tools and services to overcome the challenges we faced.

A

Hello: everyone welcome to today's sessions on embrace demo practice to ml public development.

A

My name is tommy lee, I'm a software developer from ibm code, a stands for center for open source data and ai technologies, and we also have ehong hwang my teammate, also working on the same project from ibm.

A

So today we'll go over our ml problem development and how we embrace a type of products on top of it.

A

So first we'll go over like the agenda, so we'll go over some background. Why we need to develop ammo pipeline platforms and then we'll go over our development lifecycle and some of the challenges we face and at the end, we'll go over like some of the cicd and abram we have made on top of our projects.

A

So, to start with some background, we want to automate the lifecycle of machine learning and at the beginning we have a machine learning problem where we need to take some data train it with machine learning and have some outcomes.

A

In reality, the actual ml workflow actually can spam into multiple teams.

A

So we will have a data engineer to handle the data collections and pass on to data scientists to do data, ingest data processing and model training, and we also have like production engineer that handles the training part as well on cloud and then deploy the models and let the application developers to use those models and, at the end of the day, the feedback from application developers could also feedback to how we collect data and how we ingest data back to data science and their engineering team.

A

So when we looked closer at each teams, uh each of their steps actually break down into multiple smaller steps. Let's say for data preps, you could do data cleansing data, ingestion, data, analysis, transformation, validation and splitting, and the same thing applies to like model creations where you need to build a model and optimize it validate it and then train at scales and from the production engineer, perspective you'll have to deploy server, and then you have to create like monitoring and then a future improvement and feed those feedback back to like data preparation steps.

A

So, as you can see, each of this step could be we run and we do multiple times within teams and this process could be very repetitive over time. So this is why having a pipeline system will really benefit for multiple teams and they can also share their knowledge across other teams within organizations.

A

This is why we're really interested in the q4 pipeline project q4 pipeline is a open source project that is building and deploying portable scalable machine learning. Workflows on kubernetes preferred pattern. It can. There is implementation on ml tasks, so it actually allows us to put each test into components and we use it and you could use on any runtime framework and data type and, most importantly, could also attach, with kubernetes object, like volume in sequence, so make it very cloud native and from a data science perspective.

A

uh They could use what the giveaway problem called the python dsl to create a sequence of steps and the steps could be defined with like data dependency on inputs and outputs. So you could create like a dag on the right hand, side, which is a direct acyclic graph.

A

And then, once you create the pipelines, you go this way schedule to run this pipeline periodically or as the chrome drops, and one of the beautiful things we do for pipeline is that everything could be written in a python sdk, where you only have to define a decorator for the pipelines and then defines a list of parameters and steps, then it actually compiles to the that you see, on the right hand, corner.

A

However, at the time we look at default, pipelines only supports the backend engine called argo, where we actually find some problem when running it, on non-docker runtimes and on openshift, because at the time, a lot of the ibm teams already using tacton as these devops engines. That's why we have the interest to move g4 pipelines to run on tecton as well.

A

So here what we have done is I will actually build a compiler that actually take the existing kfp sdk and compile into a tecton pipelines we're actually able to pass it to the uh modifier kifo pipeline api server that we have built and, in the background, the q4 pipeline. Api server is supposed to handle the job history and the scheduling for us, but the actual public execution will be handled by the tecton engine on the kubernetes cluster.

A

The benefit of doing this is that we have teams that optimize, tecton and ibm clouds and by doing it on q4 pipe language, we use all the plugable components from the existing truffle pipelines, communities.

A

So now, let's go over some background for the tecton project. The tecton parliament projects provide kubernetes style resources for the currency, ict style pipelines, title introduced. Few new crts, like test pipeline test, run and pipeline runs and pipeline run, usually defines as a execution of the pipelines where pipeline is a template, how all the tests are composed together and then we have test run that actually handles the execution of each individual task, and tasks could be as simple as just compiling code, running tests or building and deploying images.

A

So when we look at how to map tactile back to the kubernetes terminologies, essentially each task is equal to one part on capability and every step inside a task is an additional container inside the path, and this step is to be very useful later on when we actually try to do metadata tracking on q4 pipelines.

A

Now I want to go over why we want to run ml workflows on top of q4 pipelines. Well, the benefit is like ufo pipeline provides metadata trackings, where in this case, metadata would be an artifacts, a other fact. In ammo workflow sometimes could be a data, it could be models or it could just be metrics and having metadata tracking is very beneficial because we could find out which data a model was trained on and then use that to compare the previous model runs and then further on.

A

We could carry on states from previous models and we use previous computer outputs as well.

A

So this is what all the fast tracking looks like on q4 pipelines. Here we are using a tfx taxi trip examples and after running the pipeline, you will see the above all the fact are generated.

A

Each artefact will have a few columns of metadata information.

A

There are pipelines, name id type, uri and greater app pipeline is the pipeline that generate these artifacts name is the name of the artifacts id? Is the unique id for each artifacts type in this case? Is the tfs type for each artifact you generate?

A

Uri is the object path where the artifact object is stored and created at is a timestamp of when this artifact is created.

A

When you click on each individual artifacts, you will see more information below where it shows the uri name pipeline name who is producing it and the state of the artifacts and next to it you will see a linear explorer looks like this linux explorer will. Let you view the history and version of your artifacts in this case, artifact could be anything you produce from the task, for example, model data and metrics.

A

So we'll take a look at these examples. The statistics test at the beginning, we produce a schema, gen artifacts and once it's produced, the schema test will consume that schema, join artifacts to actually put a few more other effects called transform example, validator and trainer, and then you can see like transformed artifacts is actually further being used by four other tasks and then so on and so forth.

A

You could see more details when you click on each task, and this is one of the reason why q4 pipeline is very useful for data scientists and we really want to enable this on top of tecton as well, because techno doesn't support discount artifacts tracking by default. We actually create our own solution to run tecton pipeline on q4 pipelines to actually let us track the artifacts.

A

So what we have done is when we generate a tecton pipeline. We actually give like some artifacts annotations, so the q4 pipeline api server will able to based on those annotations to inject a post-processing steps.

A

In this case, we will copy the artifacts in the post-processing steps and actually push to the default pattern, object store, and in the process of that, we also have a metadata writer to actually track each part and get the information on where the artifact is stored and how we want to store the other effects.

A

Then those information will be stored into mmd, which is aml metadata service on qflow and it's backed by a relational db as well, and once those information is in lmd, then give u I'll be able to use the mmd api call to get those information displayed in the graphic ui fashions.

A

So, with the new artifact tracking mechanism, we introduced, on top of techton we're able to replicate the exact same q4 public experience for examples, the ui, dag the workflow and the metadata trackings.

A

But everything is backed by tektonyamo right now, and this is very beneficial because we are able to give data science the same experience while maintaining tecton as our core engine for running all the pipeline and develop services.

A

Now I will process to you hong, to talk about how to apply developer practices to our q4 problem with tactile project.

B

Hi everyone, my name, is young thanks, tommy for working through the project that we were working on now. You know one thing or two about the kill flow pipeline with tecton. Let me share the journey about how we've applied default practice to enable the ci cd for nightly build release.

B

I will talk about the tool and service we use in just a bit. We know that devops helps you on various aspects, from development releasing to production.

B

So first thing: first, the scope: we define the scope where and what we want to implement and evaluate if the sizing and efforts are feasible and tangible, our minimum viable product is to run a set of end-to-end email pipelines using cool flow point files with tecton we'd like to achieve two goals here. First, is quality assurance the ci for pr and commit only verifies the unit test, so we like to use nightly view to verify some end to end ml pipeline scenarios in order to make sure the code quality.

B

Secondly, we want to automate the nightly view releasing by pushing views when the end-to-end scenario passed once we had clear scope, I would break it down to smaller tasks, including retrieving source code, compile and build individual components. Cluster configuration deploy, components and services around predefined ml pipelines, verify the result published in nightly views, resource, cleanup and etc.

B

Each task should be self-contained, and some tasks may depend on other tasks for the coupe cluster. It takes time to spin up and tear down, so we decided to exclude leads in our initial phase. Instead, we will use a pre-configured good cluster to run our cicd pipeline repeatedly.

B

We will handle infrastructure as code later. Besides variants of ml pipeline scenarios, many different operation tasks like some ml pipeline require unique production, roll out or canon redeployment. Some may need their own configuration and application monitoring. So this reminds us that flexibility scalability are very important when designing the cicd pipeline.

B

Those also influence tools and service that we choose for the implementation. Next key role is the ml pipelines when applying ci cd to ml pipeline development. There are some challenges that we need to deal with. Like I mentioned earlier, ml pipeline has its own production rule mechanism, for example canary rollout, with accuracy, monitoring or model serving with microservice architecture, so combining it with development cycle. It makes the csd pipeline complicated and not easy to maintain or expand. On the other hand, ml pipeline scenarios may change as the project moves forward.

B

It may need an extra step for data processing or model merging after the hyper parameter tuning, not to mention mostly the ml pipeline, are lindsey and have many dependencies.

B

Based on these characteristics, we realize the cicd pipeline will be better of having features like a it can manage, sequential and parallel task execution within the pipeline, be easy to integrate with heterogeneous systems c a pipeline monitoring. So based on our observation and experience, these features would be influential in the adaption process.

B

We did some research according to the scope and the challenge, as I mentioned in previous two slides, we found ibm tool chain can fulfill most of our needs. A dual chain is a set of tool, integrations that support development deployment and operation tasks. It also provides lots of tutorials with templates for various applications.

B

For example, microservice on kubernetes are progressive throughout your application, in kubernetes and etc. With the toolchain service, you can choose tecton as the delivery pipeline engines for kubernetes application development and that's exactly what we need. Also the tektung task. Template catalog provide a lot of tasks that we can use to compose the cicd pipeline directly.

B

For example, affection could configuration cub context, scripting could control deployment, docker view, docker, sign a code risk, analyzer scanners, git report operations or post message to select channel or invoke a script to execute testing and also devops insights, integration, to record view, testing and deploying records and more so. The key feature here is both development and devops teams use the same language to compose the ml pipelines and also ci cicd pipelines.

B

It helps both teams to sync up their needs and changes.

B

Here is the details of the cicd pipeline that we set up for the m2m ml5 verification by using the toolchain service check the pipeline flow on the right hand, side. It starts from code retrieval or followed by build process for each components and services.

B

Then it trigger unit test along with docker view and docker push to internal container registry. Then the deployment process starts components and services are deployed to good cluster, and here comes the ml pipeline execution, as well as resolve verification.

B

After cleaning up the resources, the ci cd pipeline ends at nightly view. Releasing each of the box represents an individual cicd task. Overlapped boxes run concurrently. Each of the tasks waits for the previous tasks. To finish, a task can also use the execution results from previous tasks in our cicd pipeline. If any of the tasks fails, the pipeline fails.

B

Some of the tasks may encounter intermediate value. We set up the retry counter and to avoid this kind of situation ruins the whole pipeline execution and waste of time. Basically, you can leverage all the features from cactus to compose your cicd pipeline. In our case, we only use a few of them in the middle of these tasks. We also insert the slack notification task to send out progress to the slack channel for those who are interested in the ci cd pipeline. They can easily get the status update by checking the slack channel.

B

Also a view test and deploy status are sent to devop insight services. It help us to analyze the overall pipeline execution.

B

As you can see in our journey, we leveraged the curated task templates as a set of building blocks to program our cicd pipeline again, both ml pipeline and cicd pipeline use, techta share the same syntax and use the same features and both kind of pipeline are stored in the same repository.

B

Meanwhile, developers and devops engineers can check the pipeline execution status, file, select channel view test and deploy status are published to develop insight service. It automatically provides basic quality and risk analysis information from there. We can define specific policies to determine if the bill could be deployed to the production or not system. Integration becomes easy, breezy the nightly view releasing is fully automatic as well.

B

Now it's just the start of the journey. We still want to cover more specs in our ci cd pipeline, unlike infrastructure configuration and production rollout on the right hand, side, I highlight the develop coverage we have so far. We believe with ibm tool, chain, service and tech tongue delivery pipeline. We can accomplish more operation in no time finger crossed now. Let me pass it back to tommy.

B

He will share some future experiments. We'd like to adopt.

A

Thanks ian for explaining the cicd process, so because you can see the demo process for our project, it's very complicated.

A

So that's why we want to convert that as code to simplify our ci cd process and what the experiment we have done over here is try to utilize the cap sdk, because we have built a compiler able to convert q4 power and sdk into a tecton pipelines.

A

We're thinking can we actually bring those pipelines into the managed developer services that actually runs uh tactile android.

A

In the background, the result is possible for simple pipelines or we're actually creating some challenges when we put it into actual depth of practice and because q4 public dss is mainly designed for ml pipelines, not all the features from tactile and argo exposed to dsls and that actually creates some challenges because, like a lot of the existing tectonic community task template where we call it catalogs for building like devops are not really one-to-one mapping to kfb dsl, because it's not all feature exposed to the dsl.

A

So that's why it actually creates like a very high learning curve for the existing tactile developers and they cannot define their normal workflow using some of the type of features because they are not exposed to the kfp dsl.

A

One of the possibilities we'll learn from these challenges is that maybe in the future, when tactile gets more mature, we could design a dsl, that's specific for the tactile platform rather than relying on the dsl on g4 pipeline. We asked dependencies on running on q4 pipelines,.

A

So with this uh we're going to end the sessions- and here are the resources on the browser we have built, you can see the github link for problem with tecton's tactile projects and the device services we have used, and you can also see some of the tutorial two chain catalog and techno task, catalog we use for our depth of services.

A

Thank you very much.