Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 1 Dec 2017

Previous Meeting

⏯

youtube image

►

From YouTube: Towards Kubeflow 1.0, Bringing a Cloud Native Platform For ML to Kubernetes - David Aronchick

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Towards Kubeflow 1.0, Bringing a Cloud Native Platform For ML to Kubernetes - David Aronchick, Microsoft & Jeremy Lewi, Google

In December of 2017, a small number of folks from a handful of companies introduced Kubeflow; an open, cloud native platform for machine learning. The project has gained a lot of momentum with hundreds of committers, thousands of commits and stars. With the traction, many companies are asking if can Kubeflow help them bring their ML practices to the next level, and where Kubeflow is going next. This talk will discuss the growth of the Kubeflow ecosystem and its place in the lifecycle of ML development. We will provide concrete examples of how Kubeflow is developing new applications such as Katib for hyperparameter tuning and Kubeflow pipelines to address gaps in the landscape. We will also show how we are using Kubernetes and Cloud Native technologies to glue these applications into a cohesive platform, and where Kubeflow will be going next.

https://sched.co/MPax

A

Hi everyone, thank you. So much for coming. I am David Oren check, I, lead, open source machine learning and ni lofts at Azure and I'm, one of the cofounders of cube flow and very pleased to be presenting with one of the other co-founders of cube flow I've, never presented with him before Jeremy, lui and Google. Wasn't that crazy? We were scheduled for a talk. I, of course, was very responsible. Didn't show up for the talk, so he gave it on his own. He knows a bit about cube flow, so that was ok.

A

So thank you so much for the opportunity to talk this is about towards 1.0. It's like Zeno's paradox we're almost there we'll keep getting there we're not there. Yet, who knows what this is? Anyone want to think? Yes, no nothing! This is the snack. It is the first machine learning thing in the world invented in 1951.

A

Machine learning has been around a while and the reality is. You know you look at something like this right in the year 2000, it was just pipe Python, basically, I mean. Obviously there were other machine learning frameworks, other machine learning tools, but really things started to accelerate about the year 2000 2006 it was numpy 2007, there was TN, o pandas scikit-learn in 2010 2013 cafe was introduced, 2014 deep learning for Java 2015 was a Cambrian explosion of frameworks, a bunch of stuff coming out, and now today you have even more than that and that's fantastic.

A

You see more production-ready machine learning, deep learning coming out than ever before, and so you may say, we need one more ml solution, because that slide isn't full enough, but the reality is is that everything you saw on that slide? It's really focused on solving a very specific problem, and that is how you build a model, but when you talk about bringing things to production and actually having an impact on your business, this is what you actually see it.

A

You know it takes just a few weeks to build a model, and you know just about a year later, it's still not in production and here's a real-world example there's an amazing repo out there for github natural language search in just two weeks. They were able to demo with a Jupiter notebook.

A

Then three days they built a mock-up and it's ready to go and then three months later still not in production, because you know it's hard and and like I said, the problem is this: everyone in our industry tends to focus on this as being the key issue, but the reality is it's just a small component of an overall machine learning, lifecycle and and you're like wait. A second have I heard this before you know, isn't machine, isn't the idea of building many micro-services and tying them together a known thing and you're right?

A

It is that's why you're at a kubernetes conference, that's talking about nothing but micro services and all these kind of things- and this is a slide literally from four years ago, co-founder of kubernetes Joe betta, talking about how everything at Google runs inside a container and the way that we were able to bring all these micro services in this design.

A

Pattern to the world was through something like kubernetes and cloud native applications and that the idea that you can deploy all these applications all these many elements of a workflow or a pipeline, loosely couple them together and then give you something that is larger than the individual parts. So the question is: can we use kubernetes to fix this?

A

Can we use kubernetes to help provide a framework in which you can coordinate all these many services together that bring your machine learning to production and you can, except, if you want to do that, you have to learn all this stuff right, which is not so good. I promise, you I know a lot of data scientists and trying to sit down with them and talk about gang scheduling, or you know. The right parameters for kube control will not make anyone happy.

A

So the answer is: could we work towards something like cloud native ml and obviously we think we can? So what would cloud native ml look like? Well? First, you might select a set of applications that you would use for your overall machine learning pipeline. Just a few examples up there. Then you would describe some of the scenarios areas that you would be trying to achieve with those machine learning tools, and then you would have a number of different infrastructure components that combine together that you would use to provision and actually execute those things.

A

But the unfortunate part today is you end up doing something like this in your test. Matrices and everything like is loosely coupled together and it looks like a nightmare. So that's why, with Jeremy and fish who's standing on stage they're in the picture in December of 2017, we introduced cube flow and the idea was to make it easy for everyone everywhere to develop, deploy and manage portable distributed ml on kubernetes.

A

Taking the advances and all the thoughts that we had up to, you know kubernetes and cube flow excuse me cube flow or kubernetes and cloud native apps and bring them to ml via cube flow. So what that looks like is this right, the same stuff? You have before same tools, we're not asking you to learn anything new, we're not even asking you to learn any new procedures or change your compute. What we want to do is give you a common platform so that all those components can connect to cube flow inside of cube flow.

A

You can use the same standard, tooling and patterns and practices that you were using before, but then you're able to roll it out into the same compute environment you had before so. The timeline looks like this: we introduced it in December of 2017 and every 4 months. We've had a cadence are accepting every three months. Every quarter we have a cadence over a new release. You see they're cute flow 0.1. We had our go and ambassador and Selden making it easy to get in make it easy to serve maybe easy to begin orchestrating.

A

We started adding things like hyper parameter, Tooting additional frameworks in chief of 0.2 0.3. We added our updated our TF job and made it easier to deploy through KF control in Q plus 0.4. We had a very significant release with the introduction of cube flow pipelines, which we'll be demoing today and at Q flow 50.5 we're going to announce some brand-new things around Ferenc, which is a way to execute from your notebook. A very rich experience and then also our brand new jupiter web app, which makes it very easy to deploy.

A

So with all that, we think that cube flow is a cloud native platform for ml and our tenants are its composable, its scalable and it's portable, and that means it is designed to run your applications in the way that you want anywhere. That kubernetes is, can work. The architecture as a whole looks something like this. You can see there, you have an ingress endpoint that is native to kubernetes, and it can connect you to either a rich dashboard.

A

Let you execute individual operators that are part of the queue flow platform or let you serve your models, but underlying all of that is a rich orchestration through queue flow pipelines and then, additionally, rich metadata, allowing you to coordinate between all these various steps. In the past year, we've had an incredible amount of momentum. You can see here just a few of the many companies that are contributing and- and you know when I started, I could never have hoped for the kind of velocity of real-world contributions in the past 28 days.

A

We have over 500 pull requests and 80 unique authors and, and by unique authors, I mean this write like a lot of people, think about kubernetes and they think, like. Oh, that's just a Google thing, but the reality is you, don't fill a 10000 person organization with just Google right, there's Google or not Google. So what is cute flow? Look like cute flow a year and a half old, no funding. No anything! You know this is what it looks like you have Google and you have not Google, and that is tremendous.

A

That's what we mean when we say community driven like these are things that are solving real people's problems today and there aren't just like little one offs of people contributing this is serious stuff and to show you how much progress you've made it in this time.

A

The items on the left are what we launched with, and the items on the right are, your you know: standard critical user journey, you're able to ingest, transform, train analyze, your data hyper parameter, sweep you name it it's all built in, and it's all orchestrated with cue flow pipe and, like I said we have an enormous amount of contributions and I cannot think enough.

A

All the folks up up here who are really pushing the for platform forward and really making the business there on so, like I, said we're now at Q, fo 0.5 very proud to announce that we have a bunch of notebook improvements. We know that we want to give everyone a rich notebook experience without having to leave without having to you know our joke. Is we don't not date, a scientist to even have to use letter K if we can get there? We're gonna be happy. Second, around deployment.

A

Part of what we want to achieve with Q flow is being completely portable. So, whether or not that means running on your laptop running in cloud, one cloud to the CLI, the deployment experience is the same and you're able to use all the features of whatever environment you're deploying to and then finally pipelines again. This is one of the things that we feel very strongly is the differentiator is the thing that's going to help bring data science to production faster, is orchestration through pipelines and we're added GPU support.

A

We added a number of tf-x integration, that's tensorflow extended from google and then some upgrades are an external support. So with that I've done a lot of talking, let me hand it off to Jeremy who's actually going to show you all a show off some of the features in 0.5.

B

Thank you very much David. So what we're going to be showing you today with a demo? Is we really want to show you how you can go from developing a notebook developing a model inside a notebook all the way to deploying that model into production and the key idea that we want to show is that we're gonna?

B

Let our data scientists be data scientists, because they're gonna be able to develop their model inside the notebook, using the tools that are familiar with them, and then we're going to be able to let them easily scale out by training as on kubernetes and also deploying on kubernetes and then finally, we're going to end up building a CI CD pipeline to actually deliver that into production using coop flow pipelines, and so we're gonna make our devops folks and SR ease hoppy, because we're gonna stick to the to the best practices for DevOps, because we're gonna follow.

B

You know get-ups workflow and be entirely declarative. So let's get started with the demo. So this is the coop flow central dashboard. So this makes it really easy for our data scientists to navigate among all the different applications that are in coop globe. So we have really three today, so we have notebooks sorry I, clicked on the docs, so we have notebooks. We have cat tip, which is our hyper parameter tuning system and then, of course, we have pipelines and so we're going to start off today by using notebooks.

B

So if we go to the notebooks dashboard, you can see that if we make it super easy for each user to have multiple notebooks, so you can have one running, you know PI torch and one running tensor flow. So they can. You know let one run one job and then start another one to do more work. It's super easy to launch a new notebook, so they can just click there to launch a new server.

B

All they have to do is type the name for their notebook and then they specify the doctor image they want to use, and so we ship a bunch of stock, runtime environments for tensorflow and other frameworks. But you can also build you know: custom Jupiter, docker images to have custom environments, and then you can also customize the amount, the resources and even attach extra data volumes.

B

So it's super easy to attach like an NFS share using PVCs, so you can easily share data I'm, the different notebooks and other pods in your workflow, so in this case I'm going to go ahead and connect to an existing notebook that I have running so here's my notebook and so in this notebook we're going to be building a model using extra boost to predict housing prices and so I will be using the Eames data set, which is one of the standard sort of data sets used in the machine learning community for benchmarking and testing.

B

And so the idea here is to predict housing prices using features like population area, income, etc, and so in here we've defined a cell and we just defined the code for training the model. Is that legible? Or should I zoom in zoom in.

B

How's that good okay, so we've defined the functions for for training our model and so we're just calling a extra boost here and I pushed that code, four out of the way into a separate file just to make a little bit more legible. But you could put the code directly here as well and then also we have our predict function right. So this is normal Python code that people with data scientists would normally write in the cell or rubber. They would like to write it.

B

We've just asked them to put a little bit of structure around that code and define that as class methods. So we think that's very. That should be very doable for data scientists, because they're already pretty familiar and comfortable with Python.

B

So now what we can do is we can go down here and we can instantiate this class and call training, and so, in this case, training is just running inside our notebook and similarly, once we've done that we can go ahead and call project local predict, so we're just gonna load up some data and then call predict and so we're generating some predictions. So this is all running inside the notebook. So this is the normal sort of workflow of a data.

B

Scientist would be developing and iterating on another model inside a notebook, but you can imagine now that they want to scale out.

B

So maybe they've only been you know, testing out their model on a small sample of data because they wanted to train quickly and now they'd like to scale up and maybe run on the full data set, and so now they want to let it run you know, maybe they want to let it run overnight or even longer, or maybe they want to scale up and use more resources, more CPU and more GPUs, and they want to.

B

Maybe would like to run that as a batch job on kubernetes, or maybe they want to even you know, run multiple jobs in parallel because they want to do some. You know like a parameter sweet so with coop flow and using fairing, which is a high level library that we've built in coop flow for using and consuming kubernetes. We make it super easy to take this notebook and launch it as a job on kubernetes. So all they have to do is define really two variables.

B

The first is the docker registry, where they want to push the docker images that we're going to build, and the second is the runtime docker image they want to use as the runtime environment, and so we're gonna use the same docker image that we're running our notebook in and this way we get the same exact environment in our batch jobs that we're using in our notebook. So here we can go ahead and with just you know about two lines of Python code.

B

We can go ahead and build our doctorate image directly from our notebook and you'll notice that it's super fast and we think that's super important because we don't want to get it in the way of the data scientist stab test cycle. We want that to be super quick, so we've gone ahead and built the docker image and now using about two more lines of Python code from the Ferenc library. We can go ahead and deploy a job on kubernetes, which is going to go ahead and run that same training job code.

B

So it's gone ahead and launched the job and then introspect just logs. So it's doing the exact same training function, except now it's actually running as a job on kubernetes. And if you don't believe me, we can actually fetch by running coop cuddle the job spec, and we can see that we've actually gone ahead and spawned a kubernetes job.

B

So now we'd like to go ahead and deploy that using as a model and create a rest endpoint that we can begin to hit, and so we again we can do that really easily directly from the notebook just by using a fairing and so again with like two lines of faring. We can go ahead and deploy an endpoint and that's going to give us an in custer IP address that we can hit and you can see if we again check under the hood.

B

We can see that we've actually got a deployment, a kubernetes deployment that we've created using the same docker image that we've built before, and we've used Selden to wrap that predict code, Manan HTTP server, so that h, that predict function will now get invoked in response to a HTTP request and so down. Here we can go ahead and read some data and again do prediction. Only this time we're doing the prediction by sending an HTTP request to the to the server.

B

So we've gone all the way from developing our model and prototype in our model, inside a notebook to being able to launch that as a job on kubernetes to scale up for training and also to deploying that as a deployment on kubernetes, so that we actually have a production ready server.

B

So now we're going to go ahead and build a CI CD pipeline for deploying this into production using pipelines. So the way that we do, that is you to define a pipeline. You basically define a Python function, which is very gonna, define your pipeline and you annotate that with the pipeline annotation. So we just have this DSL dot pipeline and pipelines can be parameterized.

B

So you can specify what parameters or arguments those pipelines will take, and so, in this case we're going to take the path to the training data, as well as the model file where we wanna save the model and then in every step in your pipeline is defined by a containerized operation.

B

So we just instantiate that using the DSL container option and we specify the docker image that we want to run so in this case, we're just going to specify the same docker image that we built before, so that we're getting getting a consistent environment across all of our steps, training, deploying and then running in our pipeline, and so the first function in our deployment pipeline is going to be the trainer model.

B

And so we're just going to specify that we want to invoke the Train function that we defined before in our class and then we're going to pass. This specify the command-line arguments that we want to pass to that training function, which, in this case, would be the training data, as well as the model file where we want to save the model and then to create a CI CD pipeline.

B

What we're going to do is, after we train the model we're going to deploy the model, then we're going to validate the model which in this case, could just be sending a prediction to that model. And then, if that passes, what we're gonna actually do is create a PR which is going to go ahead and update that deployment back to point to the new model, and so after we do that. What we do is we use the pipeline's SDK to compile that pipeline.

B

That's just generating the Argos back to run this, and then we can just submit that pipeline by calling the run function in the Python SDK. And so, if we do that, that's going to go ahead and launch that pipeline. And so, if you click on the link down here, we can see the pipeline running it's going to take a little bit of time. So we can click over and we can see a pre pre finished pipeline, and so you can see we have the four steps that we defined before.

B

So we have trained, deployed, validate and then create PR. And then, if we click here, we can see all the runs of the this pipeline. So in cupola pipelines we organize all the different runs of a pipeline into an experiment, and so we can see all of the runs and we can see the currently running pipeline and yeah.

B

We can see that it's made it through the first three steps and it's doing the final request to create the PR, and if we go over here to get hub, we can see that we actually created a PR and what we, what this PR was doing, is we've basically updated the model specification which, in this case is specified by an environment variable to the latest model which we've saved in GCS.

B

So going back to the die.

B

So, just to recap, we went all the way from development to production, and while we did that, we made our data scientists happy because we allowed them to stay in their data, their environment- that's very friendly to them. So we allowed them to continue to develop and iterate inside a notebook, but we nonetheless allowed them to leverage kubernetes for scalability. You know they were able to do both batch jobs and scaling out for for training and then deploy the actual model.

B

But we also made our saris happy because we ended up with a declarative, get-ups based workflow for pushing this model into production and then, importantly, at no point that we have to rewrite the notebook code in order to deploy it. And so, with that I'm gonna hand it back to David.

A

Thanks Jeremy, so I hope you can see that you know we really are thinking about this. How do we bring together both the data scientists and the asari? The the quickest way to hurt your data science /? You know productivity is to ask them to learn an entirely new set of tools and ask them to pick up a bunch of things, especially if that's hurts their ability to iterate using the tools they know and love like notebooks and visualization, and things like that.

A

On the other hand, unless you build a system like this, which is get ops based, which is ml ops base that understands how to transform those artifacts produced by data scientists in a production ready to a production ready, artifact you're gonna have nothing but trouble so 40.6.

A

We know that you know there's still a lot of stuff to do. One of the top requests we get by a mile is multi user support. We want to make it very easy for multiple people to join together in a single cube flow deployment, yet still use isolation between them and we're gonna be doing that. We're also have sto and bringing a number of our api's to 1.0 a second. We want to really make double down our investments in metadata.

A

We think that tying all these experiments together through rich metadata, you know, using known schemas and and making it available for everyone excuse me to use- will be really really powerful. If you can describe all the steps you just saw, Jeremy walked through there with rich metadata and with rich tracking. That makes a huge difference. A second thing that we're investing in in that same capacity around tooling is we will be swapping out case Annette for customized case net was the decision we made when we first launched the project.

A

It is deprecated and we're moving towards customized, which is a upstream templating system in kubernetes and we're much more confident that it will be maintained, long term and then finally, around pipelines, we are going to continue to do our pipeline work. We consider that to be critical to building that production, ready, end-to-end machine learning system. So what does this must be to talk about one email, so I have good news.

A

All these bits are already production, ready, I, don't know what you're complaining about, but but that's the truth right. These things kubernetes and tensorflow and pi torch and tf-x ISTE Oh Selden. These are all core components in cube flow. They already run in a containerized way. They already run in a way that that your SR ease and your data scientists can trust massive massive deployments inside Google in the real world.

A

You know are already using them today and- and that's really part of the good news about get flow, we're not going out there and introducing a brand new machine learning system and we're not just arbitrarily declaring something as 1.0, because you know somebody needs it. We know what needs to be production ready, and that is when you start your tensorflow job, it better finish or it better give you a good reason why and and that's what kubernetes and cube flow is designed to provide already.

A

However, you're right cube flow as a whole is not 1.0, and by that we mean we really want to be thoughtful about when we declare that 1.0, because that means we're going to snap to AP is that we plan on supporting for a long time and in order to do that in order to really feel comfortable.

A

We want to make sure these critical user journeys are absolutely nailed, and that is you know exactly like you saw there, building and training from your notebook, making sure that multiple people can share a single, a single cube flow deployment, having a uniform experience for installation. Unfortunately, today, right now we don't have enough people testing, we don't get enough bugs from from different cloud environments and so on, and we need to start detecting those so that we can make sure that we really do support every cloud that is out there.

A

We want to make sure that the pipeline's get richer and and use the best practices of get ops and we're gonna be doing a bunch of work around that and then finally I you know one of the biggest areas of machine learning is tracking and and reproducibility, and we're gonna be doing a number of investments around that as well and- and we don't think we need to be perfect about all of these things by the time we get to 1.0, but we want to make sure that we have an experience that makes it more powerful for a data scientist we don't want to have to have them take a step back in order to adopt this platform.

A

That is production-ready.

A

One additional thing that we want to talk about is governance. This is an extremely important issue. If a project doesn't have people who are not from a different company on the governance board, it is not an open project. I cannot stress that enough, and that is something a big step. We're taking. We we have a proposal for governance right now. We would love for you to come, join. Give us feedback join the governance join our SIG's. This is an extremely open project.

A

If it is a single company driving the boat, we have done not done our job. I have not done my job, so please come talk to me, talk to any of the people and we will figure out how to make this work together.

A

You know, I was there for the original governance of kubernetes and we very much are following off their lead and I always like to end with this slide, because I'm full of a room of very smart people who are thinking about machine learning and in the reality, as as data science will touch every industry, and it's up to us. The people in this room, the people on the feed to think about the people outside this room, the people who are not watching the feed. How can we make their lives better?

A

How can we help them be smarter about their jobs? How can we help them change the world through tooling and techniques and making it simple for them to bring their knowledge to the to the fore like I said. Cube flow is open, open design, open community, open chat and lots of links. Please come to any or all of these you can. We have couplet all right. We have a slack. We have a weekly community meeting.

A

We have you know hundreds of people here who are engaged in to flow today and we're happy to help you you have me, you can email me at any time and I will have happy to answer your cue flow questions and with that, thank you very much.

A

So we have about, we have about four minutes left any questions. Yes, tail.

A

You got it so uh tans point was that we have an hour and a half long showcase at right. Now, after this talk at the Google booth for could flow birds of other please come by I will be there. Jeremy will be there we're ready to answer your questions. Oh sorry, it is at the Google cloud booth. Please come yes.

C

A

Yeah, so the question was one platform is extensibility: do we plan to support a standard API for extensibility for a number of different toolings? The answer is yes and no right now. The idea is that the packages, the way you deploy a package is fairly standard right. You can do basically, whatever you want, but you can easily go. You can grab a package and it will deploy in a fairly well-known way. Part of what we're doing around metadata is giving you standards for writing out what your package is doing and things like that.

A

But the reason I say yes, on the other hand, is when we see enough people doing an implementation. The community has started to form standards around those particular implementations. So you look at something like serving. We have a brand-new effort called KF serving queue flow serving, which is an effort to standardize on that API and that's an API. That's not just gonna be standard for packages that the plain cute flow, but Google and Microsoft and other folks Blumberg, are all contributing to that as well. So that's how we standardize the API for serving.

A

Similarly, some of the folks from ant financial and others are standardizing on the operator framework for jobs, for example right now that we have a PI torch job, an MX net job, a tensorflow job, excuse jobs, so on and so forth. We're like! Oh wow, we've done this a bunch of times. Let's come up with a standard for that, so it is not we're not going to be standards driven, but we are gonna say when there's enough of them out there, hey, really we should get together and let's move forward did Jeremy anything.

A

So how much effort do we put on open to making it run on open shift? We do have a number of Red Hat contributors right now and our p0 is, if you have a kubernetes conformant cluster, it should run so if it does not run well you're on us and we are gonna, take that bug and make it sure it works. That said, I would love more particularly usability. Testing set up testing debugging.

A

We don't have as many OpenShift contributors as I would like. That's not bad-mouthing anyone, it's just what people showed up with and if you have openshift and you want to run it and file 100 bugs we would love it. We just don't have that automated test matrices as much as I would like.

A

So the question was: do we have any approximate date for queue flow 1.0? The answer is when it's ready? No, the answer is many components will go 1.0 fairly soon right, so TF job is just about 1.0 pi torch job is about 1.0 and we'll make you know we're trying to make good decisions around that I. Don't I would feel uncomfortable, giving you an absolute date. I would say if you need it to be.

A

1.0 come talk to me and let's figure out what you're concerned about, because I can tell you many of the api's are feeling pretty good right now, but there are a lot of brand-new things like metadata and things like that that just need miles on the road. So the best you could do is help us by just taking it and running it and if it doesn't feel right, then that's come time to come.

A

Talk to us, but the more feedback we get about it running the better we are but like I said it really is the component to production-ready. It is being used in thousands of places today and and even if it's just that individual component, it runs great yeah.

C

A

Yeah, so the question is: do you have to use notebooks to do all your cute flow stuff? Absolutely not! Thank you very much for the segue in an hour and 55 minutes to 255 what time is it 2 hours and 55 minutes I will be presenting an ml ops presentation about using cue flow and ml ops. In that I won't note, no books, I will do it only with Python scripts.

A

The reality is is that pipelines is very, very rich as long as you can containerize your code and, in fact, under under the hood, what Jeremy showed today, even though he was driven from a notebook that was all Python as well, so there's absolutely nothing tying it to that you're able to containerize your Python script, give it standard end points and then put it in a pipeline and it'll run. I. Think.

B

For example, to develop locally in your favorite IDE and then deploy things and scale things out on to your cluster. So that's another common use case for it.

A

What's that oh and we added functions as well, any other questions.

C

A

Yeah, so that's a really interesting question. The question was: how do we deal with versioning of the data and obviously you know artifact tracking and things like that. That is something we've heard. A lot of right. The realities is that the the Data Platform you provide is ultimately probably going to have to do some form of checkpointing or artifact production. Because of the nature of you know how you version data over time.

A

Cube flow will pick up that information and allow you to store it in things like you know your overall metadata store and things like that, but I, don't think, is a first class thing. Cube flow is gonna solve that. That's it, for example, pachyderm has a rich data solution and they are integrated into cube flow today, and so that that's an example where you would use pachyderm pachyderm would provide you a checkpoint and a version data.

A

You would store that metadata and then you would consume that in the next stage of your pipeline, but but we wouldn't expect Hublot to solve that natively. It would expect versioning from the underlying data provider yeah.

B

And then the integration point would be like the metadata store, so, like you could with snapshot your volume. For example, a rip dough has a solution for snaps, rubbing your your PVC volumes, and then you can record that in your metadata, so you can deco I ran training and here's the the snapshot that I ran on. So now you have a completely reproducible pipeline for yourself.

A

Sorry, Mineo or Divya.

C

A

Yeah, so you should talk to the gentleman right right ahead of you who, from Eric doe, because they are doing a lot of that multi cloud. Multi deployment, storage issues again, that's an area where we would hope that below the layer that would be provided we we're trying to have fairly clean abstraction layers between the infrastructure, components, the kubernetes components and then the cube flow components we want. You know we're not going to try and stall storage we're not going to try and stall networking. These are very hard problems.

A

We're gonna rely on the platform that we're running on in specifically kubernetes to solve some of these other issues.

A

Any other questions we'll be at the like I said: Google Cloud booth doing a birds of a feather right now, and then we have oh yeah. Thank you. We have a number of other key flow talks and you know people walking around cue flow shirts on come talk to us. Thank you over.