Cloud Native Computing Foundation Kubernetes Community Days (KCD) Africa 2022, 24 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Transforming your Jupyter Notebook to a Reproducible Machine Learning Pipeline using Kale

Description

Kubeflow is an open source, cloud-native MLOps platform built on top of Kubernetes. Kubeflow is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable, and scalable. Kubeflow KALE (Kubeflow Automated Pipelines Engine) enables you to deploy Jupyter Notebooks that are running on your laptop or in the cloud to Kubeflow Pipelines seamlessly. In this session, we will convert a Jupyter notebook based on customer churn prediction into a Kubeflow pipeline using Kale. Our goal is to build and deploy a reproducible customer churn pipeline using Kubeflow Kale.

---
KCD Africa 2022 is the 2nd iteration of the Kubernetes Community Days Africa, a CNCF-powered free community event. Visit https://kcdafrica.com for more information.

A

Welcome back everyone. um Thank you for sticking with us to this moment. We have two more sessions left and uh next, we'll be having lady joseph and he'll be talking about transforming your jupiter notebook to a reproducible machine learning pipeline using q. That sounds interesting and yep. Let's bring all id up.

A

Hi oladi, can you hear me.

A

No, I can't hear you, let me see.

B

Hello can hear me.

A

Yes, I can hear you now.

A

um Yeah, it's not appearing. It's muted.

B

A

I think it's better now.

B

A

Yeah, so sorry, let me introduce you first now. So uh joseph is a machine learning engineer with the current focus on enterprise ml ops. He currently works as an independent community as well stands for cloud-native emerald office organizations. He has worked with mlm organizations like maven code uh and ericto outside of machine learning. Joseph comes from an energy background, air language, that's a big voice. His interest in energy lies in the adoption of technology to foster the energy efficient operations. Finally, joseph likes playing games and listening to dancehall music. That's interesting!

A

Maybe we should talk more about the dancehall music after the session go back to you, joseph.

B

Okay, thank you very much, um so I'm going over transforming your jupiter notebook to reproducible machine learning pipeline using kill, and I look forward to like um this session with you guys. So uh my name is joseph already. Like the price poster said, um I currently work as a contractor. Mlaps consultant with um eric. Do so. I would say the cloud natives um mlabs company um with products like you and your likes so I'll be working, not sure how the kill platform can be used to create um reproducible machine.

C

Learning pipelines.

B

So, to start with, um let me just go over what this session is for and if you are here, um the section is for like data scientist and devops with little experience with kill flow and what's what this session is about, so we're going to cover so depending on time. So the plan is covered, just a digit, recognizer notebook and you tell code notebook so depending on time. The focus funnel is on digital recognizer.

B

So we're going to look at look out for keyframe notebooks, specifically I'm jupiter lab. They were going to look at the kg extension with q4 pipelines, so what this section is not going to cover so this session is an in-depth machine learning or kubernetes course, and this session is very python programming course.

B

So it's about the notebook will be converting to represent machine pipeline. So it's the popular um mist or the digit recognizer um notebook it's on cargo for everyone, so I mean everyone can have access to these so about the notebook. So the question of this notebook um was actually sponsored by my organization and it's an open source notebook, so you can actually have access to this notebook. While I go over this um session, so if you can see this is the link, um github.com killflow examples traymaster then under the digit recognition cargo competition.

B

So I think I'll just wait for five seconds for people to just look out this link, just the keyflow um organization under the keep you go to the examples, repo under examples right where you go to the digit recognition, cargo, competition directory or folder so prep site for this course. um This leads to knowledge about cloud computing environments like aws, gcp or azure, basic understanding of qualitative, architectures and kubernetes concepts like pods controllers, nodes, container images and volumes, etc so familiarity with ml concepts like um algorithms.

B

I expect those are attending the session to have like a good knowledge of a machine machine. Any workflow I mean any machining workflow. You should be able to like read your pandas um to read the csv data you should be able to like um do future transformations, you're able to do a program, that's name without modal drain and the light. So um the bonus key for the service bonus introduction to keep flow fundamentals- that's like I'm.

B

By doing so, um let's get skillful overview, so to start with, so obviously just give me something of what kubernetes is all about and why um kale or chips okay is like bit on top of kubernetes. So one of the key benefits of cheap to kill leverage from kubernetes is the portability microservices and scaling.

B

So when we talk about portability, we're talking about um being able to like um create a program somewhere and being able to, like we've, been able to reproduce the same program elsewhere to not having to like have environmental environment issues or dependency issues, so with microservices workflowcon is like interacting multiple services and which kubernetes like communities offers services. So we have scaling so when it comes to skilling, we're talking about I've been able to like scale up your computing resources or use to scale down your continuous depending on your workload.

B

So this is like a um a brief overview of what um of what kubernetes tends to offer. If you kill okay, let's talk about cheap shoe itself and what keeps you doing so keeper is an open source project at the convergence of cloud native and email. So keyframes is a mlx platform for data science and operations team.

B

It's a complete toolkit for an ml workflow, including data model training, parameter tuning module, 7 and module monitoring, so keyframe was launched by google in 2017 started as uh it started at google as a way to like externalize. The tensorflow extend experience tfx and it is apache 2.0 licensed what challenges that does key for himself.

B

So, basically, keep trying to solve um m select, make your deployment of machine learning workflow on kubernetes, simple, portable and scalable, but the idea is that we know that kubernetes has this orchestration power, this portability power and this auto scaling power already. So um how can we leverage our parts like? um So how can we leverage this part to make our deployment of machine learning workflows very easy? So this, which is where q2 comes into place, so keto makes that data loading, verification, split in processing, feature engineering, a parameter, tuning, opponent, observation and monitoring.

B

Very, very easy. So we're going to go over um the different um tools and how keep flow is like the um all-in-one um platform to do this and why it's like um it's best to use a keyframe. So, um basically, typically your model building, you you did it on jupiter, you did a database rcd or visual studio code. Keyframe offers you all this all in one. So if you want to do your model training, you probably use python scaling. The tensorflow keys also offers this to you.

B

If you want to do a parameter thing, you probably use cigar determining ai or any skill. If it also does this from auto ml analysis, ml pipelines, metadata tracking and deployment, so keyframe offers all this all in one. So it's just like your only one mls pipeline.

B

So so we talked about this kit for architecture. So basically, in this um session, this workshop we're going to um use the following components and kill flow. We're going to use the jupiter network, we're going to use kit for pipelines, we're going to use operators and we'll be looking at the central dashboard so to go about um nodes broker. Basically, qrs has a notebook feature which gives you the same. Jupyter notebook-like feel um the same jupiter um you have in your local computer, the same group collab like field.

B

You have in your cloud internet and ide. So the people offers you the same set of um features.

B

So we have, um we have jupiter lab rcd and visual studio code embedded in kubeflow notebook server, so you can run any of these um ides to carry out your machine learning workflows using keyframes books.

B

So um we all have an idea of what jupiter lab is. um Hello. Was my property notebook for jupiter lab? So I won't really go in details. What is all about so.

C

It's basically just a web-based environment.

B

For creating jupiter networks, that's it so! I'm jupiter learning cube flow so to go to the pipeline basics. So um the the focus of this session is talk about um representable machine output, representing machine learning pipeline. So before I go into how to create represent machine learning pipelines. What exactly is right on this pipeline itself?

B

C

B

A description of an ml workflow that includes all of the components in the workflow now the combined form of an execution graph.

B

So basically, a pipeline is basically like um your end-to-end uh machine learning process as tags or steps, for example, if you would like, if I um okay, so if you like, like create your ml, if you'd like to go through a typical model development cycle, it's called machine learning development cycle, you're, probably going to start from like your data injection or you call it data loading and after data already, you probably want to do like future transformation or you you could want to like clean the data after that id feature transformation after feature transformation.

B

You want to do model training after model training programs parameter afterwards or you could do a parameter while doing a model training depending on which, whichever works for you and you want to do like model seven last the model monitoring so in keep the pipeline. You get to define all these things as steps.

B

So, at the end of the day, when you, um when you define all these different steps, it comes out in form of like an execution graph but they're connected to each other. So before so, for your data loading, um you could before you get ceo, you start from your data loading from your data loading. You get to your training data step from the clean data step. You get your future transformation step from the future transformation step.

B

You get to your model, training step and after your model, training step you get to your model, seven steps, depending on whichever workflow you would like to um build. So crypto pipelines offers you this uh feature.

B

So in q4 pipeline, each steps are defined as component pipeline components. What is the pipeline component? Basically, a perfect component is because a set of codes designed to perform a particular task, so apartment company, an example of a company, could be a data loading component, so whatever that loading company does is just um just performs data loading operations, an example of a pipeline um component is a model training component. What the model training company does is it carries out module training? That's all!

B

So um pipeline combinations are self-contained sense of user code package over the dog has documents and they perform one step in the pipeline, so components may run severely or in parallel, based on the way these pipelines are defined and we'll get see all these. In the answering example, so crypto pipeline design good, we have orchestration experimentation and reasonability.

B

So just like experience all this in the pipeline, I think you can see what's time, yeah.

B

Okay, so what's included in the key flow pipeline ui, so you have like all things included in keyframes. We have like a ui, so the ui um is similar to what is up at my right hand here, um whereby you have like your own notebook pencil boards modules. Snapchat volumes experiments pipeline runs recurring, rather access and execution.

B

So, with the help of sdks we are able to, uh we have problem q flow sdks. We are able to actually create pipelines that can run that can run and appear in the uis.

B

um Okay I already talked about is the component packaging component. This is.

C

B

Of a cube through pipeline, like just like an example, so you can see this is um this is for the chicago taxi trip data sets it's a popular update as I've used in most google tutorials. So basically, um this is like data injection process and there's the panda transform data frame into a csv format, there's the removed format. So these are all these things that are checked out all um steps or components by the performing particular attacks or the performance operation. So basically, what hd boost train does like it model? What hgboos predict does?

B

Yes, it predicts on an xbox model. What calculate the regression matrix from c as we do, is basically, let's calculate regression matrix, so we get to actually see all these things are live in the technical session. So let me check this cl4 pro cap users for 10 minutes get them to go experiments.

B

um I'm communicating, can you hear me.

B

So basically, this is um experiments, keep playing experiments. Basically, an experiment is a workspace where you can try different configuration of pipelines for some of us uh um familiar with mlops we have, and for those that are not familiar with them and I'll try to like just um explain these things so.

C

B

An experiment is like whereby you, when you like, typically let's say you- you train a model with um certain parameters. Let's say I try and model with um is equal to five b is equal to six and c is equal to ten. So the moment I change the moment. I change it to like, as equal to seven is equal to nine and six goes to like twenty, and I run it like I've generated a new experiment.

B

So basically an experiment is where you you're able to like implement new set of ideas, and you see kept you get to keep the pipeline walk through the same and you get like um different results. An example here is, like um you can see this image, so it is an open version um project, so different um experiments were created for the same setup of workflow.

B

So we have like pipeline run, so basically iran is a single execution of the pipeline. So the moment you are done compiling once you are done creating your pipeline and you compile and run it. So you get a single a pipeline run. So a pipeline run is a single execution of your pipeline.

B

So all these things that it might seem like a little bit overwhelming at the moment um you get to see it hands-on with this simple um digital organizer of those books, so recurring browsing, recording, you're trying to like um repeat it, particularly let's say, like you run it like last week or or let's see, I tried like schedule, um run, let's say: okay, um I want the run to like perform weekly.

B

So with the requirement feature, um I guess like uh implement this, and this is actually very important in continuous training um projects or continuous training um workflows. Let's say um you expect your data sets. You expect new set of data sets coming weekly, so I mean, with the help of the recovery run, I don't need to like recreate new notebooks or create new pipelines. Requirement. Helps me with this.

B

So these are, let's go over these, so this was. uh This is basically um a step and all you can quality pipeline component so it performs. This is the load data step. We process data, step, model, training, step, motor evaluation, step.

B

I'll be going over the hands-on session now, so let me go over I'm going over the amazon session now so, first of all, before you can actually create this type of pipeline. You need to have cheap flow installed and this there are different ways to install keep flow. You can install keep your via package distribution, you can stock youtube via manifest or you can install keep your bare keyframe as a service.

B

um This is what I use like I'm trained, my kid for pipelines or my kill pipelines, so um I would to for those of us that are new to their fulfillment. They have no idea what keeper the service is. All about.

B

Probably you need to like just go to keep flow. um The irs.com.

B

B

So I mean you could use it to try your amazon like um sample, notebooks, that you have that you want to convert into pipelines.

B

So for me, I've done that already, and I mean it's the the notepad the website um as a very good ui, and you have the kind of explain things- will switch views like create this um free service to use to back to.

B

So they are different, like I said, um q flow um packaging for distribution, so either you install queue flow on your os. You install your keyboard, customize obs inside on google cloud, azure or other services. So the idea is that this keyframe works on the um leverages kubernetes for its operation. So before you can install clifford, you need kubernetes to be installed, so um we have an aks azir combination um service. We have um gke google cognizant ng.

B

Now we have eks, that's for um that's for um aws, okay, um if you go to the kill flow documentary or if you go to, if you go to the key for documentation, um I think it's quite easy. I don't give a documentation website by that, so I could just go with this. Give me documentation- and I find myself here on that documentation. I'll probably click on installing queue flow and it should be able to. um You should be able to get me started.

B

Let me see, I think this was directing me installing kill flow so yeah. These are like different ways. You can install qfs. There are different doors that can guide you through um installing kit. For let's say you don't want to use the keypad the service um platform, you can go through this.

B

So this is keyframe 7, which I'll be using. Now, um I think, let me just go back and just sign myself um start with cube, slow.

B

Yeah lunch is for free.

B

um If you have your system with you, you can probably watch this process as well. So if you don't have an account you can sign up, so um we have an account.

B

Next then, here this should work pursuit code, so I'm coming out. I'd have to like check this check. My phone.

A

C

B

I guess it's so let me just get off.

B

Verify receive a voice call.

B

Instead, okay, I think I've got a voice call.

B

B

Okay, so um I just got a call from the code.

B

So if you can, um can you guys can I guess you hear me.

B

um So can you guys hear me.

B

B

Okay, thank you so um once you've created the q plus the service, um so it brings you here. All you need to do is create a new clip for deployment. So I do have something running I mean the website is um quite explanatory and it should you should be able to like um create this. So currently I have like um seven days left so keep flow as kicked out. It gives me like um two weeks free of um this, like I mean try my pipeline.

B

Oh so my username is user and my password is my password, so you guys have to like um you also get yours and you log in so once you do that it brings it to like um you keep flow, it brings it to your keyflow um ui and I think I would like to open this in a new called youtube page. It doesn't temporaries right.

B

So I can still see the ui.

B

Okay, so um this is the notebook, so I do have a notebook server created. So what I'll just do is I'll go to the process and create a new notebook server. So I'm going to delete the one I had before I'm going to delete this and we're going to do some experiments. I had before running to create space. Okay, I think it's late. Now, let me do some snapshots. I had um yes, there's gonna be some snapshots. I had.

B

B

um This book so currently on um a new notebook.

B

So my name, let me just call it um kcd, um so I'm using jupiter lab, so I'm probably going to leave it at jupiter lobby house use jupiter about cs jupiter, I'm versus visual studio, color, probably like jupiter I'll, probably pick visual studio called r studio, whichever one works for me.

B

Okay, yeah um set my cpu requirements. um Let's say I want um two um and I want to be gram tcp because so um my rom, um I could set this to um 10 um 10 big rom and I think that's that's all see you launch.

B

So this shouldn't take up to a minute.

B

So while we wait for this to load, so I think um if you are with your laptop gig- probably just go through this with me, but you like, um since this record, will probably get get it later. So I mean I'm going to go to this website this github repo and I'm going to clone it. Let me just show you where I'm going to so this is like an open source repo. I contributed to this and yeah.

B

So what I want to work on is the kill jupiter, the kill of these recognizer notebooks, I'm just going to pull on this repo copy link address this. Is it okay, just launch wind connect.

B

um Somewhere connecting I'm going to open it tell me now: let's open it, tell me now so.

C

This is like the same.

B

Jupiter environment you get um from your local computer, so git clone.

B

B

So I'll I'll go to I'll go to um the digit recognition, notebook, which is this now open, the key notebook which is this everything. So this is, um do you recognize that kill pipeline so I'll? Just I would walk us through um the notebooks. It's basically just a markdown that just talks about what competition is all about and yeah, so to be able to create these pipelines and that's like um so basically this notebook server. I have the new server I created via the cluster. It's just like an empty environment.

B

It doesn't have any um it just has some just python environments and stuff, so it doesn't have some libraries installed. So I need to go through the process of um installing some libraries I would want to use and the library that I would want to use for this notebook.

B

I I would need tensorflow c bond w get and pandas.

B

Since, if that says that's the case, um let me work through it. So I'm going to install I've saved it in the requirements.txt file, so I'm going to install it.

B

I'm going to pass the flag, doesn't see all those standard, outputs.

B

So after installing um the it should take, probably two minutes select install after installing I'm going to like um import all the necessary libraries so to now walk with you and you want to keep your kill. This is like where the magic um comes into play and I would like I would like it if, um if you love paying attention before, I think just probably just as much as I can say, I'm concentrating here so um while I'm waiting for this to install I'll I'll click on this kill extension here.

B

So if you haven't, if you are using um the keyframes, the service um um listing cluster you'd, have the keyflow pipeline deployment panel or the kill deployment panel here. So you just click on it. You click on it. It opens you enable the pipe you enable the panel so after enabling your panel, you have just in case if you've not created an experiment before you click on new experiment, you're passing the experiment name in my case it is digit, recognizer kill.

B

The pipelining meets the digit, recognizer kill as well, so the description it performs, processing, training and the right training and prediction of digits.

B

So um the moment I enable the kill deployment panel, um you see like the notebook changes, you are seeing some tags there like skip imports, um pipeline parameters and the likes I'm going to just explain what these things are. So the the beauty of um kale is that it actually um gives you the opportunity to like create components straight up from your notebook like you, don't need to write down extra codes or you don't need to um do anything um extra. You just need to enable the kill department panel create an experiment come here.

B

You see this like um this. um This pencil sign, you click on it.

B

It tells you what kind of cell should this be so basically, since I've already installed on the libraries, I didn't need to pass this into my keyframe pipeline again, because I don't need to. I don't want to always um install this libraries every time I every time I um run the pipeline since it has been installed already. My notebook server kell is going to pick it up and you don't have to stress yourself about it's always installing every time you want to create a pipeline.

B

So since I've installed, I'm going to have to skip the cell so to skip this just you click on this and you can skip cell from the word. Skip cell just keep the notebook. So don't um don't perform this operation just keep it then I'm going to have to import my um libraries. So let's make it a habit to ensure, like all your imports are in one place. So it makes skill like be able to like um understand like some of the libraries better.

B

So when you, when you call them or you want to, you- want to use them down the line in your machine learning development um process. So basically um you click on this. You pass in all your imports. You click on import from the word imports. Just everything you're trying to import into the node just pass it here then, for my pipeline um parameters, so pipeline parameters are basically um parameters that the pipeline uses or several components can use. For example, now I could use the batch size in multiple places.

B

So if I was quite I've passed it as a pipeline parameter, so several components can use my pipeline parameter. It's like a global variable, no in python.

B

So basically I pass my pipeline parameters just like the parameter I want my. um I want my pipeline to always um have access to every component. You always have access to this. If, if I just use this, if I don't say I just want it to be there, so my public parameters are.

C

B

My upper parameters, so I have like learning rates for shots for lr epochs batch size, um convolutional dimension, one convolutional dimension two, so I passed that so I also set like um random seed, but in this case I started to like skip it. So I like I, don't want this to go into my um pipeline um graph. Imagine like me having a graph that says: like okay, cetera, don't see like it's, not really, um it doesn't really look nice. I mean.

C

You could do it if.

B

You want to, but I didn't just I didn't, do it it's a choice, so I skipped the cell. So that's not good, like creating a pipeline step itself. Everything we've been talking so far have been things like. I've been adding 12 skills. Let's talk about the codes itself that perform these steps, which is the pipeline component, so to create a pipeline completely kill. It's very, very easy, and it's as easy as this, you have like specific commands like the perform download data.

B

So what I want to do then I'll download this data from my git up from the kiplo git of um repo I pass in the data link, I import the big get and zip file. I come to this pencil button. Click on it. I pass in the step name, which is download data.

B

So when you click on okay, I click on pipeline step say it tells me to pass in the step name and it's dependency, so basically when what it means by dependencies, yes, which other steps is this um step depending on, since this is the first step in my uh pipeline workflow, it doesn't depend on anything. So I just leave that please down so um this is like my my source code that downloads um this and the trend, data test and the sample submission data.

B

So um in case, if you are wondering now, the source code is about you can just check. This is the source code here from this github repo um you can play around with it and you can create new ones yourself. So this is so cool everything that you're seeing me doing is is in this kit of repo for your use so um yeah. This is the download data step. So next stop is the load data step.

B

So, okay, after download data, I want to load data into csv pandas um unload this download the time to like um load the safety time so pandas as it did, and as a data loads it and output as a data frame.

B

So to do that I'll click on this pencil sign I'll come here. Click on pipeline step, passing the step name, load data dependency. It depends on download data. So what is just means like I mean it's going to load what the download data has downloaded.

B

So you see why um this this kind of interconnected anyway, so I mean the download that doesn't penetrate any downloads data, so that's like an injection process. Typically, if you're working in an enterprise like what you want to do is like um there's, usually like storage, probably like um aws buckets as your blob storage, your google cloud are buckets. Well, let's say: there's a feature store somewhere that um that you load this data from more like a data ingestion process.

B

So this low data depends on what the download data downloads. So I mean I read the data frame, for I read the data, the csv data, from what the data download data step and download it, and this is it so the beauty of kell is that multiple nodes cells can can have a single um multiple node boxes can have a single pipeline step. It can be under a single pipeline, for example. This the case here is a is the load data step. If I call me this is also a delay data step.

B

This is also the load data step. If you can read this, it says: leave the step name empty, to match the cell to step to data. So basically you can make multiple cells see a single pipeline step, so the next next um pipeline type is we process data? So it's as easy as I'll just come into this page. So I can click on this. Go to pipeline step. Click on pipeline step come to the step. Name enter pre-processed data, its dependencies so to be processed data.

B

I need what has been loaded, so it depends on the load. Data it doesn't depend on. Download data depends on the load data, so below data information gets passed. The process data, the key process, data step mixes up the divided data like carrier transformations- and you know so then, after the pre-processed data. It's time for me to do my modeling so for the modeling, so the modeling you you come here just like I did previously. You come to the cell type.

B

You click on pipeline step come to the step name, you click um you type in modeling or whatever, with a model name of your chair whatsoever. You feel whatever, if you like, giving it. It depends on the pre-processed data, so the preprocessed data gets passed into the model instead so yeah. What I just did I create like my model: it's a tensorflow, um it's a compositional model.

B

So this is um my modeling step. I have several duke sales passed under my moodle. Instead.

B

This is my compilation because of time I set it to like. um I was I'm chipped above epochs I set it to okay. I set it to just two epochs.

B

Take note so, basically what I just did in the mo for my for the parameters I pass into the pipeline parameters before I pass them into my new cells. I I tell the notebook what type of data type it is in this case, while I was compiling my learning, lanyard is of float, is of data type floats and for some of us, um if we are of python background, know what fluid is or yeah. We know what float is in general, so this is like the decimals, so my linearity is of type floats.

B

My my batch sizes of type integer, my epoch, is of type integer, so I just pass it day and the notebook understands this so the.

A

B

Step after modeling is the prediction step, so I just try to make this as um simple as possible, and you know when it's time for the key and years can ask questions. So the next step is the prediction um step, so the prediction step depends on the modeling.

B

So basically, what has been passed to the patreon step is the model that is the mood that has been fit has been fitted already, there's a fitted module and I make it out of the twitter module to like carry out predictions and after my predictions I get confusion matrix out of these. um I believe we all have an idea of what of your matrix is all about after that um yeah. So I just um plot my confusion, matrix so I'll be able to visualize it in the ui.

B

So that's all about the uh that's our about creation of pipeline. Let's see this um coming to actualization. So once you've done all this, this is also the submission code. I have, I skipped it because I'm not submitting anything to anyway, so you can just keep this cell so compile when you're done just click on compile and run.

B

So what kill? Does it validates the notebook and it checks if your codes are correct? Once you see it's checked out, your codes are correct. It takes snapshots so basically this option. What kill offers you like once you um once you've created once you've created the notebook, it saves all the information, all the working environments, the storage inside your notebook server. It says it inside kill in the event. Okay, let's say you want to use it next month. You want to use it in two months time.

B

It's just this information is ready, backed up for you, so once you're on um once you're done, you get this view wherever you can deal with your pipeline, how your pipeline is running.

B

B

So I want this to actually appear like on to view their jaw pipeline. I think this is.

B

B

So to feed the actual pipelines, what it does it creates. um It creates um a volume. First, after creating the volume, you see, the download data step.

B

So download data step is running so when downloader step is done, you're going to get the load data step when the data step is done, you're going to get the pre-processed data step, 2 data step is done, you're going to get the modeling data step when the modern data step is done, you're going to get the prediction data, which is the final step- and you have your pipeline um graph.

B

B

So once I see these bluetooth issues like the um issues the pipeline is now um running actually download data step is now running.

B

And this should take about 10 minutes um in total, so I think um I can afford questions now. What will wait for on this run to like I'm finished? So? Basically, the idea of this pipeline is that, like I mean you, you want this process. uh It says it's built on that kubernetes um in the event that, like you're training, something with a very huge wall cloud like um it has the auto scaling feature because it leverages on kubernetes like to scale up based on computing resources and business computing needs.

B

Another feature of this pipeline is that, let's say you want to perform a continuous training operation for some of us that are from mla program for someone that are trying to get into like an mls background. um You can create a pipeline that goes into like you can create a pipeline that the output is the model that goes into a model registry, so, depending on how you would- um and this is like enterprise envelopes. So the download data step is completed.

B

And where this is no details there, what will for the load data step as well?

B

C

Like, as I was saying,.

B

Probably, the output of your pipeline typically probably is a model that goes into a model registry and then, thankfully, for okay, so far parameter tuning features. However, the um the focus of this um session is not on a parameter or model saving or you like just on pipeline creation, possibly like similar time talk about um model 7 with kill or q fuel talk about a parameter, a parameter tuning using k or q.

B

So if you have a question like we can start asking now before we're done, department have been um put up completed. I think that's like that's all. If you have a question guys asking no.

B

Hello, sorry, I'm guessing.

A

Yes, we can hear you, I think we have a question. Is there a kill extension for collab.

B

Okay, so um collab is not built on kubernetes nobody's on top of kubernetes, so that would be possible.

B

So there's no kill extension on column.

A

C

I think there's another question on how you can join the community, although I'm not sure if his case um kcd community or the new community.

B

Okay, so um the kill community is.

B

So I feel, like um typically there's a key flow slug, um probably just google. I think let me see if I can google this um keeps with slide channel um channel I'm in already, so I just hope it doesn't direct me to what I mean. Okay, so here's the keyboard slab channel, you could google these and the first thing that comes up is communicating um flow click on it and it should direct you to how you should be able to join the official keyframe um slap channel.

B

um I hope this is helpful, so this is like get um all of this information. There.

B

um Let's, let's go to the pipeline, so the load data step is completed.

B

We're going to present you just deaf questions or.

B

So, at the end of the day, your pipeline should look like something like this, which we would get to see. So you happen to like something like this. Something like this.

B

You have like the create volume, download data load data, it process, data, modeling prediction.

B

So, let's let me get get parties. This is it so, while this is um okay process, data is completed so modeling second to the last pronunciation.

B

Let me copy this, so you could see the visuals live.

B

um So if you can hear me I'm trying to like login um to like my private browser, so um the display could like come out in full. So if you guaran I'm going to stop sharing now I'm going to like request for access there, so I mean everyone could see like a full pipeline.

B

A

It's coming up: okay,.

B

Okay yeah so like this is the full pipeline.

B

So the beauty of of kill again is that you could actually see some of the things you're doing there. No visualization, okay, you can see some of the things you're doing like.

B

Or something: okay, yeah, for example, some of the sorry it's it's! You know myself.

A

Alrighty, if you are speaking, I'm not sure we can hear you.

B

The um output of your notebook here, just as it is in your just as it is in your notebook like when I call the dot ad operation yeah that admit, thought it's still the same thing I'm going to see here when I call it dot, shape method. It's still the same thing I'm going to see here and the modern operation. I think I should see the module training step. Probably when it's done, you would get to see it. So it's just your entire notebook.

B

Like I mean you don't get to like stress over or I need to like do some things like turn this into a pipeline like it's your entire book, the same way you have it in your collab saying what you have with your gpt level, you just upload it here and turn it to a pipeline.

B

So, typically, if you would um do this with the um cubeso sdk, the traditional key vsd key, you would need to like write some components individually and you need to like um run some methods, create clients compile- and unlike this whatever I just have my notebook. I don't stress about that, and I need to just click things type things quick things type things up.

B

Yeah I have a pipeline and pipeline is actually very critical when you're um when you want like reproducible ml workflow, because every time I run this, I'm always going to get the steps like every time I run I'm just going to get steps. I could take this. um I could take this step on on somebody's uh and someone's like keep flow cluster different from my I'm still going to get the same thing, I could pass the output of my step to like a modal registry.

B

It's going to be the same thing so like uh this is like this: the beauty of machine learning on kubernetes, uh we machine learning, leveraging kubernetes, orchestration ability, scanner abilities or capabilities and um the portability and capabilities. So this is like my the output of my from my notebook like the model. The summary is like the training step and the last one is supposed to like show me my confusion matrix.

B

So there are still other um features which I didn't get to like show in today's session. um There are ways in which you could like output like um pipeline parameters for for some of us onto the experiment tracking like he wants like we have to compare different than pipeline matrix. um Let's say um initially around my first experiment, I got like 90 and I'm not running it, I'm getting like 89 percent.

B

I want to be able to like track these changes, so these are some of the features you can also like get from like your pipelines and keep you in general.

B

So let me see if I have something uh training: the past. Okay, let me go to experiment so.

C

B

Are this is like the run ui, so these are some of my futures um experiments that passed a field experiment.

B

um Let me see experiment, good experiments.

B

So, basically, you get to see um the matrix here. You get to be able to like compare these different metrics and it gets like we have to like rerun the matrix, the desired um the desired pipeline. Let's say you run about 10 different pipelines and you you want to pick the best um one days on your own.

B

These are some of the things that um you don't like taking into consideration, but that's not like I'm the scope of this um session, so I won't be going over that, but that's some of the things you could do with. Would you like experiment tracking? You could do like um applying a pipeline orchestration with teach. You could do model seven with key flow. You could run distributed, training with kill flow.

C

And yeah, I think that's.

B

It so we have our pipeline run completed, so that's how to locate the pipe and machine learning pipeline using kill.

B

That's all this is the visualization.

B

This is my confusion, matrix.

C

All right, um thank you, alrighty for that awesome section. If you have any questions for. Ladies, so far, you can either drop it in the chats or you can also check him up on his social media, handles where you can ask for the questions, just in case you're, unable to get a response here.