Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 22 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tutorial: Introduction to Kubeflow Pipelines - Michelle Casbon, Dan Sanche, Dan Anghel

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Tutorial: Introduction to Kubeflow Pipelines - Michelle Casbon, Dan Sanche, Dan Anghel, & Michal Zylinski, Google

In this session, you will learn how to install and use Kubeflow Pipelines to create a full machine learning application on Kubernetes. Starting with an empty environment, you will create a Kubernetes cluster and install Kubeflow from scratch. Then you will build and run a full pipeline that first trains a model using TensorFlow, then serves the model, and finally deploys a web front-end for interacting with the resulting predictions. You will then move into a notebook to build and run your pipeline using the Python SDK. You will become familiar with Google Cloud Platform tools such as Cloud Shell and Kubernetes Engine. Prerequisite: fundamental knowledge of Kubernetes. Setup: must bring own laptop. Qwiklab/GCP credits will be provided. Note: this session showcases Kubeflow features newly released since the Seattle workshop.

https://sched.co/MPgr

A

Welcome everyone you're here for the introduction to cue flow pipeline. Well, we're gonna do two get together. Today. It's build an application. It'll be a machine learning application, that's built on kubernetes. Everyone should have a Google cloud account. Does everyone have a slip of paper in front of them with credentials? If you don't, please raise your hand. Okay,.

A

If you've never used the GCB platform before it should be fairly easy to follow along. But if, at any point you have questions, just raise your hand and one of our TAS will come help you out. So my name is Michelle Kasbah I'm, an engineer at Google on the cue flow team and I've been there for about a year and a half.

A

So to give you a sense of where I'm coming from before I joined, Google I was head of data science at a start-up in San Francisco before that I was a data science engineer and for about a decade before that. I was a data engineer in various enterprise companies. So that's why I'm background I'm coming from a sort of engineering data, engineering, slash data science background and the people that I have with me today we have a ton of cuteville expertise concentrated in this room.

A

We have Abhishek he's based in Sunnyvale, he's the engineering manager for the Q flow team. We have Dan Sanchi he's based in Seattle another cute law, engineer, Jeremy levy, he's one of the project, co-founders they're in the back, and the tech lead Dan on gel based in Paris, also an engineer and me how Solinsky he's based in Poland. So if you have any questions along the way feel free to ask to ask one of them. So what is cupola? What are we here to talk about?

A

You've probably heard of it before it's a framework for running machine learning, frameworks on the kubernetes framework. You get the idea, so more precisely queue flow is a curated set of compatible tools and artifacts that lays a foundation for running production, machine learning, apps didn't Able's consistency across deployments by providing kubernetes object, templates that bring together disparate, so a few housekeeping items before we get started. We're gonna be talking about a few different layers. We have our infrastructure layer, that's GCP.

A

We were platform layer, that's cute flow that runs on top of GCP, and then we have our application layer and today, we'll be learning about we'll, be building an app that can summarize github issues so throw out on a few of the slides I'll be talking about each one of those layers in the sack and you'll be able to tell which one because it'll be highlighted up at the top corner.

A

The cue flow is completely open source. It's on github, here's the link down here, just cute little slash, cube flow, and it's not just it's not just the code, that's open, but the community is open, we're open to new design directions, we're open to ideas. So, if you'd like to get involved, there are a ton of ways to do that. We have weekly community meetings. Those are at different times so, depending on where you're located in the world, you can find a time to attend. You can interact directly and github, creating issues and PRS.

A

We have a slack team and a mailing list. We have a dev summit that we're planning we usually have those twice a year. We've had two of them already so stay tuned towards the end of the year. We'll, hopefully have something planned, but if you'd like to get involved, your mother welcome to okay. So this is an important link if everyone could navigate there right now. This is the cute little pipelines, github issue summarization lab. So if you find that link, it will give us a nice.

A

Walkthrough it'll give us all of the code that we need all the instructions and some explanations for each step as we go along now all walkthrough that up here, so everyone can follow along. But if you have any questions, just raise your hand if you get stuck at any point. So what are we going to be doing we'll start by setting up our environment and then we'll create a brand new cluster from scratch and we'll install all of the key flow components?

A

On top of that, and then we'll move into the queue flow pipelines, dashboard and we'll upload a pipeline and we'll run it directly from there and then we'll move into the notebook interface and we'll run the same pipeline directly from a notebook and then we'll end by cleaning or cluster okay.

A

So it's going to ask us for a zone and whenever you create your cluster when we're setting everything up the rooms kind of divided unevenly. So what I want is for half of the room to use one region in half or one zone and the other half to use the other, so I'm going to say so. Here's the line, if you're on this side of the line I want you to use Europe West, one Bravo. If you're on this side of the line, I want you to use Europe West, one Delta.

A

Okay, that's gonna be important because after we create our cluster, we're gonna be using GPUs, and if we all try and hammer the same zone, we may run out of them. So everyone on this side, your Bravo, the side, your Delta. Those are the two zones that we're gonna use so write those down if you need to, but remember those because you're going to need to select that in a bit all right, let's get started so I'm, mostly going to talk through the introduction. So I'll skip this, but this is a publicly available link.

A

You can go through this on your own for all those who are outside I'm. Sorry, there's not more capacity, but you can do this with your own account after the fact, we'll start by opening up the GCP console, if you haven't seen it before.

A

This is what it looks like.

A

So this is our main menu up here and what's going to be most important for us, is our project ID we'll need that pretty much right away, but we'll start by opening up the cloud shell, which is this icon right up here? Is everyone following along? Does everyone have their account logged in oh I should have mentioned this before I recommend when logging into your account using an incognito window, because if you already have a GCP account, they can sometimes get confused and we're covering all the costs for this workshop.

A

So you want to use the account that we gave you because we've set it up with enough quota for GPUs, and then you don't have to use your own credit, so I recommend using the account that we gave you and an incognito with a incognito window is the best way to ensure that there's no crossover.

A

Okay, can everyone see this screen cloud? Shell, okay,.

A

All right so the other important thing for this, because we are we're summarizing github issues, and that means that we're hammering the github API they have a rate limit, especially for a bunch of requests coming from a single IP. So what we need to do very first is just to set a github account token, and you can use you can use any github account, but you're essentially generating something with zero scope.

A

So all this does is authenticate you as a user, so that we're not sending anonymous requests because otherwise we'll get throttled and things will break. Oh I, don't have the password for this count, which is fine. So when you generate this it'll give you it'll, ask you the scope and you just leave all the check boxes. Blank and just generate a token and paste that token into your cloud. Shell and save it as this variable github token. So, no scope, this shouldn't have access to anything.

A

It's read-only and all it does is authenticate you as a non anonymous user.

A

Okay, so the other environment variables we're going to set are these: we want our deployment name, project, ID and so.

A

Your project, ID, is this right here you just copy and paste that.

A

And then zone is going to be Europe West one Europe West, one B Bravo and D Delta.

A

B

C

A

Does someone want to so I think what you're saying is that cloud? Shell is an opening that it's so it's spinning it's waiting. Okay, there is not anything. I can do from here. However,.

B

There's anyone.

A

Else having trouble accessing cloud- shell, ok, let's give it a few minutes. Sometimes I can take a while to provision.

A

I think started connecting.

A

Yeah you're welcome to use your local terminal. If, if you prefer that, if you're used to GCP, if you have the G cloud SDK installed, you will need the G Cloud SDK, though.

A

Okay, so we're studying a few more config settings, we're setting our project ID and our zone.

A

And the next thing that we want to do is create a bucket. These two commands will create a storage bucket that just gives us a place to put our files, while we're training.

A

Okay, our next step is to install the cute little pipelines, Python SDK,.

A

Is this large enough? Can everyone see this if you having trouble just, let me know okay, so this is just a pip install command.

A

Then it's all of the Python dependencies that we need. This will allow us to compile our pipelines from cloud. Shell.

A

And if you want, you can pin a couple of useful dashboards. I generally, do that, just so that I don't have to search through this giant menu, but kubernetes engine and storage are two of the most useful ones.

A

Okay, how's everyone doing am I going too fast, too slow to fast, okay, I'll, slow down. I'll. Give you some time to catch up. If you have specific questions, feel free to raise your hand and a TA will come to you wave at me. If you have like a more general question about queue flow that you want me to answer, while we're waiting.

A

The zones- yes.

A

So you're a blessed one Bravo is on the side deltas on the side.

A

So who here has used a JCP before okay? So that means it's new to about half the people.

A

If you have GCP questions, it's okay, to ask those who here has not used kubernetes before okay, so it might get a little rough because I'm not going over any kubernetes basics.

A

If you get lost, you can ask questions but I think the general expectation is that you understand what keeps ETL is I will give you all the commands, though, so it should be. Okay,.

A

Okay, who would like me to wait a little bit longer? Are you ready to continue? Okay, we'll wait a bit longer.

A

Okay, so our next step, there's gonna, be quite a bit of waiting time. So if it's okay with everyone, I'd like to go ahead and get the cluster feed it, it's it's! Okay, if you're a little bit behind because well you'll have plenty of time to catch up.

A

It's okay to do some of these setup steps after the next one.

A

I can't hear you what's your question: I.

D

Said in its not saying nothing happens, are.

A

You in an incognito window, it might be interfering with your other Google account. If you already have one yeah I recommend using an incognito window with your lab account.

A

Okay, so our next step is to create a cluster and we'll do that using click to deploy, which is this UI right here, and what we need is our project.

A

We can copy that from the cloud console.

A

So we add a project ID and we want to select setup endpoint later, because that takes even longer it creates an external URL. We're not going to do that today.

A

We'll use our zone again Bravo on this side, Delta on this side, and then we create our deployment. So we really just needed those three pieces of information. Our project ID set up our endpoint later and your zone.

B

D

The view jump to the same effect.

D

A

A

Oh, the yamo file. Does the ammo file do the same thing? So if you view the mo and then apply it to your cluster, I believe there's more going on. So it's using deployment manager. Jeremy, do you know if that's the same I, don't think that's a deployment manager, yeah Mille? Is it.

A

So the question is whether, if you view the mo, is that the same as clicking on create closer.

E

But you can't just apply to your cluster: it's not a kubernetes config. As Michelle said, it's a deployment manager, config.

A

Okay, so this will take about five to ten minutes, so, as Jeremy mentioned its deployment manager, which is a GCP tool that packages up everything you need, so it's not just kubernetes Hamill's, but it's also enabling api's and setting up service accounts. It's doing a lot more than just adding the cute little components. So that's what's going on behind the scenes, and normally it would set up an external URL- we're not going to do that today, because that takes an extra 10 minutes.

A

It's doing a lot of networking, but there's a lot going on behind the scenes. This is going to take a few minutes. So is everyone at this stage?

A

Ok, a few hands, raise your hand if you've gotten to this point. Okay, that's good progress and, if you're still on the setup part, you can do that at the same time as this.

A

Okay, oh I wanted to show you what it looks like in the console, so deployment manager- that's what we were just referring to and we'll be able to see a deployment here and a cluster here.

A

So it's adding nodes and eventually we'll see all of the cute little objects show up here. So that's what's happening behind the scenes.

A

Okay, while we wait for that, let's talk about why we're doing this? Why does key flow exist? So it came out of facing having faced a lot of the main challenges and building machine learning applications. One of the biggest challenges was moving from one environment to another, so forklifting code from local into dev, into test and into production.

A

That path can be fraught with error and what we needed was something that gave us portability so a way to package all those infrastructure components together, so that we could forklift our code from one place to another without actually changing anything, because there's a fundamental difference between running something locally on your laptop and running it in a data center in production and another major challenge that we faced was this year: complexity of machine learning applications.

A

So the common perception is that when we think about our applications, we tend to think about the ml components as being sort of larger than everything else and more important noticing usually reflect reality.

A

Reality is that there are a lot of other, very important components as part of the application and before you can even create your models, you need data, there's a lot involved in getting data ingesting it analyzing it cleaning it up, exploring it and separating out your training from your test set. So you can extract your features once you have your data and then you can build your models, but once you have them, you still need to validate them.

A

You want to be doing some auditing versioning of them, hopefully doing some distributed training and doing that continuously. As new data comes in those models, they live somewhere, they live inside an application, and that means you have a user interface. That means you have some business logic. If you have users you're doing some load balancing and you need some serving infrastructure that application has to live somewhere. It lives inside a platform and for a platform you need things like monitoring, logging, CI, CD, Solutions off all of the normal things that go along with maintaining software.

A

So that's a lot more than just ml, so we needed something that was more specific than just supporting the ml components of the application. So to solve the complexity problem, we needed some things that gave us composability so a way to create components that we could reuse, common things, common patterns and templates that we could reuse throughout our applications and create logical groups.

A

A maintaining such a complex system is a challenge because it's not always straightforward to even identify errors, some models, we don't even have good ways of identifying when there's things like bias that show up. So if we can identify errors, can we recover from them and prevent them from happening in the future? If we can fix those bugs, how quickly can we do that? How quickly can we go through the development process in ml apps? That's traditionally been very difficult.

A

Can we roll back to a version that didn't have the air? All of these challenges that are generally solved in the traditional software space are pretty big when it comes to ml apps, so composability helps us to shorten that development lifecycle so that we can iterate faster and fix things and automate that process so capacity planning a lot of times in machine learning applications you may not have necessarily have regular usage patterns.

A

Training has a much bigger footprint in general than serving and that sometimes look different based on when new data arrives, um but even when we have well defined usage patterns that can spike, we have users, demand can spike at times and we want to be able to use our resources efficiently.

A

So that's why we chose kubernetes, because scalability is baked into the platform it's fundamental to it, and then one of the gke features that we really like is Auto provisioning, which you'll see it's part of the lab, so you'll get to see that in action later.

A

Ok, so, let's check on our cluster, let's see how it's doing that looks good and it looks like our queue. Flow components. Are still provisioning.

A

All right, we can probably connect to it at this point.

A

So we're still installing queue flow, but we can set up our cube CTL so that we can see what's happening.

A

So this pulls down our credentials.

A

And let's see if we can connect to it, see if it's right.

A

Okay, so we have a cluster and this command right here. This sets up our contacts so that we're always looking at the chip flow namespace. So by default, everything is installed into the queue flow namespace and we don't want to use the minus end command every time.

F

A

So everything is slowly installing.

G

A

Give that just a few more minutes before we run our next.

H

H

A

Who else is, at this point, who's everyone's setup? Look roughly equivalent to this raise your hand if you're, following along approximately I.

H

A

C

A

Cute flow is not running; we can't do anything. Do you mean for the next step? So if your cluster is not up, you will need your cluster up for the next step. Yes,.

G

A

It still installing look at me a few more minutes so I remember before we had clicked to deploy I remember having to do this manually and it was a total pain like you should appreciate how nice it is to just be able to sit back and let it do all the installing for you. You probably saw some of my early code that was like very involved and very complicated, so it's very nice to be able to just sort of wait until it's done.

A

Did anyone have any questions about queue flow, anything more general that I could answer.

A

If it's something specific to your environment, um let's get it let's go to TA over just raise your hand until someone comes.

E

A

General queue flow questions or anything about pipeline. You have a lot of expertise in the room. Is that a question for me.

I

A

So the question is: what's the distinction between Q flow pipelines and tf-x pipelines, so tf-x pipelines your right to they can compile down into Q flow pipelines but yeah, so they're, two totally separate frameworks like two totally separate projects and tf-x pipelines can run on either I believe. What's the.

I

A

Flow or what yeah air flow, so Shukla pipelines uses, are go under the hood, which is like a kubernetes native version of air flow, so two separate work flow work, flow engines and tf-x pipelines is basically one layer above that so people I see who used that have non kubernetes clusters and so the run airflow over there.

A

But they don't want to change their code, and so they want it to run in a kubernetes cluster as well, whereas cupola pipelines as communities only good question, hey any more general to flow questions, if not I will keep going. Is that once.

A

Yeah so cute, the question is whether you can use people on any kubernetes cluster or only Google cloud yeah. So it's on top of vanilla kubernetes. So as long as you have a communities compliant cluster, you can use it.

J

A

Tl is what you're gonna want KFC tail. It's a CLI tool. Well, we'll go over it here, but it's essentially a command line version click to deploy as GCP. Only but KFC TL is the command line version of this, and it lets you select your platform and it gives you all the Gamble's so that, like you, um will command it essentially. Does that for you, okay,.

K

A

Has a cluster up and available that looks good. That's most people.

K

A

So we'll go ahead with the next step.

A

This is where it gets exciting. It's it's only one more setup step and then we'll be I will be able to actually use it. Okay. So this is where we set up node Auto provisioning. This is exciting because it's a brand-new gke feature that I've been waiting for for a very long time. It's still in beta. This is why we have to do it separately, it's not on by default, but we just run this command, and this will take four or five minutes and what this is doing.

A

If you haven't used auto provisioning before it will based on the resource requests inside the cluster, it will automatically provision nodes for you. Oh, this is important for us specifically because for our training job we need GPU notes and we need a lot of them and you don't want to have GPU nodes, just sort of up and running all the time. So we've added some code, there's there's a queue flow pipelines command where we essentially just request Ram and GPUs, and that turns into kubernetes yeah mole.

A

It turns into a kubernetes resource requests which auto provisioning then recognizes, and it will give us a brand new GP with exactly as much as we need and the cool thing about.

A

That is not just that we get it on demand, but that once that job is finished and there's no more demand for that resource, the node disappears and it gets decommissioned, so we're only paying for it for essentially as long as our job is running, so that's a cluster wide setting, and that means that any any resource request you send to the cluster will be. You know within the max CPU and memory limits that you give it within those limits. Your cluster size will grow and shrink as a result.

A

Okay, so that'll take a bit: let's go over a few, more slides, okay. So let's talk about some of the main goal, so you heard about all the challenges. Those are just a few there's many challenges, but the idea behind queue flow. If I could sum it up in a single sentence would be to make it easy for everyone to develop, deploy and manage portable scalable ml everywhere, and that everyone is important.

A

The idea is to make it more approachable and easier so that not just data engineers and ml engineers are the ones who can build applications. So our three main goals we have portability. We want to be able to deploy across environments and forklift our code from develop into tests into prod and not just the mo components, but really the entire stack. We want to take our entire application and move it easily from one place to another. The second major goal is scalability.

A

This answers your question about why we chose kubernetes scalability is baked in, but we use vanilla, kubernetes api s, and so the idea is that, if you have a kubernetes compliant system, you can run queue flow on top of it. But it's not just about scalability in terms of nodes. It's also. It makes your application a lot less complicated because, for example, networking you can call services by their name without having to know where they're running. That's. Why we love kubernetes?

A

The third major goal is composability is having this single unified tool, so common patterns patterns that are specific to ml applications and necessarily to anything else. We want to be able to reuse those components and pull them in pretty easily, so not just for the production version of it, but that full product lifecycle from design to development, to deployment and then iterating over bug, fixes and patches and what's important for ml, is to support specialized hardware like GPUs and TP use. What we'll be using today is a bunch of GPUs to train our model.

A

If you know what you're doing this is how you can reduce costs and improve your model performance. So again, here's that github link, that's for anyone involved in building a machine learning application, not just the data scientists or the engineers and we're currently on the ODOT 5 release. We have quarterly releases, which means you can expect to see Oh dot, six out in the summertime and, let's see yeah, so it's kubernetes native anywhere.

A

You have kubernetes, you can run it and it does this by providing custom resource definitions for things like distributed training if you're familiar with kubernetes patterns, you'll, see very familiar ideas within to flow like this concept of microservices and declaring your infrastructure managing your infrastructure declaratively.

A

So the way we move things from one environment to another is we use a templating language, it started out with customized or with case on it and we're now moving to customize, and these are essentially templating languages for kubernetes, and we have support for multiple machine learning frameworks. Tensorflow and pi torch have their own operators, but we support any sort of any Python machine learning framework all right. So, let's check on our auto provisioning to see if it completed all right. That looks good. Okay,.

A

So we should have all of our components up and ready.

A

That looks good who, whose Auto provisioning is complete.

A

Okay, is anyone just like very behind what I knew and like me to slow down, or is this pace? Okay, if you want me to slow down just raise your hand, okay,.

C

A

A

Okay, so the next step, the next step that we're going to do is connect to our central dashboard in the beginning. Whenever we installed it, we decided we wanted to set up our endpoint later. That was a decision we made to save time because we don't need an external URL, we're just going to port forward into it. So if we run this command, we can put forward directly to the service ambassador. Is it's a reverse HTTP proxy?

A

And that means that we have a single entry point into our cluster and from there we can access all of the dashboards within cute low and our own application as well. So we are we're. Gonna want to run this command.

A

And then this web preview button- this is what gives us port 8080, and here is our central dashboard.

A

Okay, who's at this point, who has their own central dashboard?

A

Okay, we'll wait a little bit we're gonna catch up. So this is this is the this? Is the main sort of single pane of glass for cute glow, so congratulations, you've just installed cute flow.

A

What's the question.

L

A

Have the service ambassador.

A

That doesn't look right. um Let's get a TA over Jimmy.

D

A

A

So that's actually a good point. The command that we issued to set up the context will always set a send us to the queue flow namespace and if you don't have that set then it'll look like you have no services, I mean that wasn't the case for him and- and you can tell because in Pantheon, if you click on workloads and services, it'll just give you a list, and so that's all namespaces.

D

A

So cloud yeah so cloud shell and in the cloud console there, one's a UI one's a CLI, but the U I should show you everything in every namespace. Unless you specifically filter it out.

A

You'll be given a Minu yourself, maybe refresh that page, these one, not one so yep now try.

B

Refreshing that.

A

Sometimes it takes a few seconds, um so what what you can do? Okay, so if anyone is having trouble accessing any of the dashboards, a good place to look is in stackdriver logging. So let me.

B

Show you what that looks.

G

K

A

If you go into the console and scroll down to logging, I.

A

Recommend pinning this one too: it comes in handy.

A

So you can filter by gke cluster operations or container and oftentimes. That will show you a warning so I don't know in in your case you're having trouble opening up the pipeline dashboard, and so that looks like an ambassador routing issue and so I would expect to see some ambassador errors in in the logs.

A

So that would be under gke container.

A

All of the logs are sent into stackdriver automatically, that's part of gke, so any errors should show up in here, and these are these- are the these are coming out of the pod logs? So you could also just view the logs for an individual pod if you're having trouble with routing that would likely be ambassador.

J

A

That might have been because the cluster wasn't finished, setting up before.

H

They ran that command.

A

So how would they sigh I'm, not sure if waiting.

A

Okay, who has their central dashboard up?

A

Okay, that's not a whole lot. We'll wait a little bit longer.

A

That's the next step. So if you're there that's good.

M

A

Do you think that could have happened because of running on a provisioning too soon, auto provisioning.

F

A

F

A

Okay, so we had a few people whose components did not deploy completely. If that's you, the recommendation is to is to delete the deployment. So let me show you how to do that unfortunately means redeploying in deployment manager. This one checkbox, all you have to do is say, delete and then that will delete everything, but don't redeploy until that disappears.

K

A

Question is where the new deployment is triggered. That's the click to deploy, so when you say create deployment, it'll create a new one of those and then also I recommend when you add Auto provisioning waiting until your cluster is up and available. So like checking to make sure you can get to the dashboard and then enabling auto provisioning just to to me to get any deadlock.

A

If it didn't deploy entirely, then yeah.

B

A

Yeah yeah, so it should be a much longer list. If you have that, then I recommend deleting the deployment and starting over.

F

A

Yeah so again, whenever you recreate your cluster, just make sure that you can access the central dashboard before you kick off. Auto provisioning.

B

A

So I'm gonna I'm gonna go ahead and move to the next step.

G

G

G

A

You seen this before this connection refused.

B

F

This is obviously which.

A

Let's the Ambassador pod.

K

H

F

Go look at services, I feel like.

C

F

L

F

See how many in there this looks almost there, except for I, think it's not completed.

J

F

There's only 20 that should be wrong third year and pipelines UI for example, and runtimes not there so you'll have to start for scratches I think the auto provisioning.

F

Maybe after that.

C

A

Right so it looks like I ran into.

B

A

Same error, so lesson learned, don't run out on provisioning too soon good thing I have this backup right here.

C

A

H

You can skip to devotees in the project setup just step to where you set the password.

A

Why is making baby mytabs.

A

I'm trying to make this larger okay, so this is a this is a cluster where I already have things running.

A

Okay, so if you can access the pipeline's dashboard who can access the pipeline's dashboard, you are the most patient people in the room. Well time.

A

So the problem was that running Auto provisioning on the cluster essentially caused the deployment of the other services to fail. So if those were not completely installed and auto provisioning was run, then it effectively shut down communications to the cluster.

C

A

C

A

So that happened to me as well: okay,.

A

Okay, so we'll keep going and what we're gonna do now is.

A

We'll compile a pipeline, so this is that pip3 command that we ran before.

A

We're downloading a Python file, that's this gh some file and inside here is a pipeline description and we'll look at this in a bit more detail in a minute, but for now we're just going to run it to generate our pipeline command. This is a compiled, compiled Argo code so that when we upload this into the pipeline's dashboard, it will actually run on top of our go. So what we have after we compile it is this tar file.

A

And we'll go into our pipelines, dashboard, which is here and we'll upload. It.

A

A

Let's try that again, I.

B

H

Know what's going on here.

A

Do you have this, you don't have 4500. Do you.

A

That was not supposed to be in the papers that got passed out.

H

All right that.

A

Explains why the cluster is called qo2 and why it just appeared.

B

A

You have do you have a cluster.

B

H

A type to cuddle.

A

Does anyone have dev star 4505.

F

A

Your hand, if you have 4505.

A

That was a yes you, you have 4505.

A

Let me grab one out of the envelope, there's no more left so many extras.

A

But those aren't gonna be good. Do you have a close already set up? You have a cluster already set up.

K

A

Yeah, that's right.

K

A

K

A

A

With one of these, does anyone have 45 ten all right? It needs that one.

K

K

B

G

Which I can't read this from.

A

B

A

All right, so it's gonna, take me a few minutes to get a cluster up and running. So who raised your hand if you have a pipelines, dashboard?

A

Okay, that's a few of you I apologize to the rest of you that had to restart from scratch.

F

A

C

A

Well, I'll talk you to running.

M

A

Okay, so we looked at using Python using our kfp SDK to generate our tar archive. That's our NGO definition! So when we upload it, we see this nice pipeline I, wonder if I still have it up. Oh there we go.

A

Okay, it'll look similar to this or you have a train and a serve step, and that's everything inside that pipeline definition. It just defines what the steps are which container to run and each of those steps.

H

A

Once we've generated that are uploaded it that's where we want to run it. That looks like this. We create an experiment for an experiment. You can have any number of runs so that you can compare between execution of pipelines. You can see what config parameters you used and what the results were.

A

These are the parameters that you're going to need to give your pipeline your project name, the github token that you generated in the beginning and then the name of your bucket.

A

So you add those here, you give it a run name. You have any number of runs within an experiment.

G

A

You click start.

G

A

Clicking on start is negative when you execute that pipeline. One of the commands inside.

E

That one flow definition is to.

A

Request resources now this particular pipeline is running your training.

F

A

You're serving it and then you're standing up a web app and so in the Train portion. We're requesting GPUs, specifically we're requesting for GPUs and I, think like 48 gigs of RAM, so we're requesting a lot of resources and with click to deploy, we're just getting a default cluster with essentially the minimum amount of resources necessary and that's two nodes. That's two pretty standard nodes, I think it's an one with eight core, so n1 standard, eight you have two of those and that basically just covers all the services. You need to run cube flow itself.

A

So when we kick off that training pipeline and we request all those resources that don't exist, that's where auto provisioning kicks in and gives us a GPU node with enough to cover the resource requests that we put out there as long as it's underneath that max which we define when we turned on auto provisioning, it should give us enough.

A

So this message right here, it'll tell us that our pipeline is in pending state because it's unschedulable, we don't have resources but what's happening behind the scenes, is that a node will automatically be added to the cluster and as soon as that's ready, then this message will disappear. That takes about three to five minutes and then you'll get into the container, creating part where it pulls the image. It has a node that it can run on. So it starts pulling the image and then it starts executing so that training step looks like this.

A

You'll see it run through it can access the GPU. So the first thing it does: is it downloads the data files it uses a checkpoint, there's actually a pretty a pretty big model, and it would take way too long to train it in just one setting so we're pulling from a checkpoint and then running this for about 15 minutes using four GPUs and then once that's done. Oh I did promise you that I would show you what that Python file looks like.

A

So this is that file that we compiled in cloud shell and then uploaded into our pipelines, dashboard and it uses the queue flow pipelines SDK. So it pulls in those libraries and it defines these different steps. So you have. These are the these are the arguments and it defines a training step.

A

It gives it a name, defines a container to use and then the command to Senden and serving step, and we want serving to run after training, and this is where requesting those GPUs we're requesting for GPUs, and this is the request that triggers Auto provisioning, Auto provisioning is just a gke feature, but we're using the Python SDK to request those resources and that will turn into the familiar yam or resource requests that you're familiar with and then web app is our final one.

A

So this is the this is the application that uses our trained model that we're serving to send in the text of a github issue and receive a prediction for what the title of that github issue should be.

G

A

One artifact of training: if you go through the training process, one of the you click on the detail. You can actually see an artifact show up and that's tensor board and if you've ever used, tensor board before this workshop doesn't cover it. But there's a ton of detail in here you get a lot of graphs on the type of statistics that were generated during the training process, so you can find out information about your model, how accurate it is for whatever it is you care about, and you get just a ton of information in there.

A

Now this is the web app. So the cool thing about ambassador when it's working when we install it correctly, is that we can just add we can add a term to our external URL. So because we were port forwarding in we had a URL that looked pretty much like this for our central dashboard and then just adding web app with the trailing slash sends us to the application that we deployed. That's this one.

A

So if this were live, we would click on populate random issue. That would give us just some random github issue would populate this box and then generating title would bring up a prediction. So what's happening behind the scenes.

A

Is that that web app is in its own PAH running on our cluster, and one of our steps was to serve the model that we just trained, and so we have a serving pot and that web app is calling just as a rest, call to our serving pot and receiving a prediction from it, and then that would show up right here.

K

A

For anyone who is able to run the pipeline.

K

A

Anyone cuz I want actually seen it run. Okay, did you? Were you able to watch auto provisioning kick in and see the cluster account the node count inside your cluster. Go up.

A

That's one of the coolest parts. I'm, sorry I didn't I wasn't able to show that live.

A

Looks like my cluster is still resonating.

A

Okay, so our next step is to run a pipeline not just from the pipeline dashboard, but from a notebook.

A

That wasn't a question: was it okay, alright, so from the central dashboard? If you click on the notebooks dashboard, that will bring you to this screen right here and you can spot a new jupiter hub instance by clicking here. You can leave all the defaults just give it a name, and what that does is it creates a brand new pod on your cluster, and that gives you your own jupiter hub environment, so it'll take a few minutes to pull down the image. We're just using the stock. Tensor flow I think 1.13 image.

J

A

So the default is tensorflow 1.13 for CPUs question.

H

So, do you anticipate their permissions and administrators able to trade new local servers and can say that.

H

A

Question is about having administrator privileges for physically than notebooks, so I have that? Actually it's on a later slide, that's something that we're working towards in the Wando release is multi-user, so some of the components apart at some of it. Don't what are you? Are you asking specifically about notebooks or queue flow in general.

M

A

M

A

The way you flow components are developed is pretty independently, so not necessarily if it's available in one component, it's not necessarily available in another I mean it's your typical open source project, so I'm not exactly sure what capabilities exist, specifically in notebooks, but I do know that that we're working towards a more consistent approach to it. So it will eventually get to that point.

B

A

There are several people, the question was: are there people running queue flow in production? Yes,.

A

Abhishek, um can you talk about any companies that are running key flow in production.

F

A

Yeah I'm not sure we're able to name names. Fortunately,.

F

I know that there are a few companies that are using Q flow for on from this is influencing using GPUs.

F

Which are again doing on-premise deployments of horns so.

A

Inference inference in production with GPUs in the retail today, infants or infants are.

A

Any more questions.

F

D

But in the in the communal, when you choose.

D

A

This, so the question is about file systems specifically from notebooks. Yes, and are you able to use anything besides so there's persistent volumes.

F

A

The response was, if you create a persistent volume from your NFS mount, you should be able to attach.

M

A

Other question.

M

So I'm in some kind of way to declare.

M

Actually, don't want transfer today to the cloud.

E

M

Furniture quests and well, we just didn't, find any way to.

B

M

Actually, keep the dyed to local bit data locally through the same cluster is.

I

J

Any way, let's do it.

M

I've seen the related github issue some time ago, but I don't know if it was.

M

A

For an on-prem cluster within pipelines is volume doesn't go, I would think so.

M

A

I I think monitoring the github issue is problem, your respect and like comment on it because the more actively we see on it, the better. So so we think there's a new feature, that's part of pipelines, but talking to the folks at erect. Oh, they have a booth here, that's probably your best bet, if not full up on github, because it is a good feature.

A

Is this a question for a TA or for me, okay,.

C

Sorry, data stream.

A

Is the data streamed or copied stream? So it's copied from a bucket copied into the bucket that you created in the beginning.

C

A

Been copied from one bucket to another within the cloud within GCS, so the commands coming from the training step that you created, but the data is moving from one GCS bucket to another. So just within the GCP Network.

A

Okay, so does anyone have access to their Jupiter husband since then, we've gotten that far all right, you're soldiering it on I'm, proud of you all right. So after you spawned your instance, you should be able to connect and it'll look just like any other Jupiter hub instance just like this, and if you open up a terminal.

A

The first thing you want to do is install the latest version of Q flow pipelines, and this is important because, even though it shifts with the distribution of Q flow components, are developed independently among different external groups and so the pipeline's they actually release much more quickly.

A

Core q flow is every three months so once per quarter, but Q flow pipelines is much more frequent, so you always want to install certain components to get the latest.

A

So if you download the notebook, I might be able to show you. Oh here.

G

We go, this is what it looks like.

A

So here's the notebook with what we're doing here and what's interesting about this- is that, instead of having a Python file that we compile locally and then upload, we can actually just run it from within our notebook and using the pipeline's SDK. We can create experiments and run them directly from our notebook. So what we do here is, instead of just our train, serve and web app steps, we're adding a new one, and then we can actually run it from within here now. All of this is on github.

A

You can look at this on your own, but this is the important command down here, so we're compiling it within our notebook and then running it right here, and this gives us a link to our running pipeline.

C

A

It looks like this, so this is how we can modify something that we've already created without having to leave our notebook environment, so you can go back and forth without having to worry about compiling things locally and transferring files.

A

Okay, we are running a little bit low on time. I have a few more slides, but before I go into those they're, mostly just on future steps for pipeline, like roadmap, are there any other questions before we do that.

A

All right, I'll go into our last few slides.

A

Okay, so I just wanted to describe some of the default components within kupo, so click to deploy gives you a default install, and this is not a complete set of components, but it is what you get by default.

A

There's a lot more that you can add on top, but we have our ingress components and there's a lot behind this I'm just going to lump that into components that give you a single entry point into your cluster that handles all the auth for you, so that you don't have to do it in anything else running inside your cluster from there you have this little dashboard that we saw you have your pipelines, dashboard, notebooks, TF job for monitoring your tensor flow jobs, tensor board, cotton, which is hyper parameter tuning.

A

So, whenever you're running a training job, you can see nice statistics lots of pretty graphs. If anyone read the Vizier paper from a few years ago, this is the open-source implementation of that, and then Argo is the. This is what powers pipelines it's the workflow engine each of those UIs has an operator behind it. So you have one for tensorflow for TF jobs, one for pi torch.

A

Then you also have Argo and cocktail operators and then, on the serving side, you have one for serving train two tensorflow models and sell them for packaging up any Python library.

A

So, what's new in this current release, so KFC TL, the person who asked me the question about installing on-prem. This binary is what you want. So it used to be just like a shell script and it's now a fully fledged go binary. So there's lots of bug fixes in the new version, but that separates out creation of the platform from installation of cube flow click.

A

Deploy did all that together, but if you already have a platform, if you already have a kubernetes instance, you can use KFC tail to just put Q flow on top of it there's a new, so we just skipped all of the endpoints but there's in cluster off so that you don't have to use an OAuth token. There are some things related to ingress that have changed and it's do was added which will eventually replace ambassador I. Think we're working on that for our next release.

G

A

Central dashboard, you got to see the nice pretty version of it too bad. It wasn't up for very long, but it's gotten an overhaul. It used to be like orange and kind of weird-looking, but I'm really happy with the changes to the dashboard. The notebooks UI via Rick tofox they're, the ones that have a booth they've done a whole lot of work on the on Jupiter hub very happy with those changes.

A

Fairing is something that's under very active development and that's a way of from your notebook, not just deploying pipelines but actually building, containers from within your notebook building, training and deploying without ever leaving your notebook environment and then Pat tube for running some hyper parameter tuning a bunch of improvements to that we've already seen pipelines that is about all I had in terms of the roadmap for the next release that should be out in the next month, or so that's ODOT, six release and that focus is on multi-user isolation.

A

So that's probably what you're looking for that's a more consistent story, so not just on a component by component basis, we're looking at replacing case on it. Since it's been deprecated that will be customized, that's that's a fairly significant effort, so it'll take a while, but we're generally moving towards a one release, which is a lot more robust for enterprise users like having smooth upgrades, rolling, restarts that kind of thing and hardening all of our api's.

A

So if there's something on here that that's missing that you'd like to see you know, let us know there are a ton of ways to get in touch or like help us build it. That's always welcome we're very open community, really great group. If you want to meet a few of us, we have a bunch of talks right after this we'll be heading over to the cute flow Boff David was just in here. I think he went to prepare for that.

A

David and Jeremy will be giving or no David and Yaron David and Jeremy's talk is tomorrow, and then there are a few other ones as well. So if you want to learn more check that and go ahead and keep the accounts, we'll get rid of them eventually, but if you want to play with that over the next few days, you know if you want to get a full cluster up and running.

A

It does take a little bit of time, but just give it a little bit of time before adding your auto provisioning and you should be able to click the rest of it. So, thanks for being patient I appreciate that everyone came today.