Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cloud Event Horizon - Ian Coffey, VMware

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Cloud Event Horizon - Ian Coffey, VMware

Cloudevents offers a useful specification which can help facilitate new patterns and innovation in several areas of tech. In this session, we will use a hands-on machine learning example to explore how Cloudevents and evented systems can work together. First we will perform an experiment by combining CloudEvents with BentoML and Tektoncd to explore the cross-section of events, machine learning models and CI/CD pipelines. Then, we will use the Kubeflow KFServing project to deploy our models inference service into a canary setup.

https://sched.co/Zep2

A

Hello hi, I'm ian. Thank you for coming to this session.

A

Today, we're going to be looking at cloud events and we're going to be looking at an event-oriented machine learning, workflow we're going to be checking out some interesting technology and bento ml and kf serving and checked in, and we're going to be using them to kind of explore cloud events and do something interesting with them in the process. Hopefully, I will show what cloud events are but more like what can use it for and also explore some pretty interesting technology.

A

First off hello, I'm ian, I work at the open source technology center at vmware, and I've worked on basically infrastructure as a service and platforms as a service and application deployment systems in some form or the other for the last 17-ish years.

A

In the past, I've worked in a lot of different places, but uh some of the more recent ones have been heroku engine yard deus and packet, and I focus primarily on machine learning nowadays, which is pretty awesome.

A

uh I thought for today we could use this objective to leverage an event-oriented machine learning workflow, and we can do that to do something useful with cloud events, because I feel like that's that's pretty much an interesting way to learn about something is to do something useful with them and cloud events as a spec is just begging to be used uh so to set some simple goals for our demonstration system, our experiment system.

A

We want to be able to experiment locally and build and deploy in our staging cluster.

A

This is just because this is the way I work as when I'm doing any sort of data science or working with uh features, or you know doing any sort of experimentation, I'm generally starting by experimenting locally and then at some point, I'm going to build and deploy, but that's going to happen in a cluster somewhere remotely and one thing that our system should also provide or is desirable, is the ability to share these train models with teammates easily so some way to like version these models and share them and have them be able to be used by colleagues in a really straightforward way is desirable and also uh you know, we need to build and deploy specific models so like not.

A

Every model that gets deployed should be built into a docker image and deployed to a canary just the ones that we really want to intentionally uh proceed with so some mechanism to handle that inherently and that should be handled from the cli for the notebook or we should be able to deploy via ci cd in a headless mode.

A

Lastly, our last simple goal is just a safe way to deploy the model api services, so some way to have something like a canary deployment where you're deploying the canary and uh it gets getting some percent of traffic, and you know we also want to take it one step further and say that we want to auto promote that canary. If, for you know, if it's getting 10 traffic for 10 minutes, let's say- and there are it's below the threshold for whatever query we want, then it should auto promote.

A

So we want to handle that as well to do that, we're going to use these components. We talked about cloud events and specifically we're going to be making use of the python sdk.

A

Bento ml is a really interesting piece of technology that allows us to have some glue here, where we can save our machine learning models and retrieve them in our ci pipeline. We can also retrieve them, though, like as a colleague we could also you know, use that as a way to keep versions of our models. So it's a pretty flexible piece of code.

A

Tecton is going to be the heart of the build system, specifically detected pipelines, but it's going to be techn triggers. That is doing a lot of the interesting work. By receiving our events, our cloud events, parsing them and creating pipeline runs based off those events.

A

Kf serving will also come into play. That is how we're actually going to boot our canary and promote it using the kf serving sdk and, of course, kf serving's built on k-native. So we get lots of good goodies for free.

A

There too, like scaling to xero and prometheus, is how we're going to do our auto promotions, we're going to await specific criteria in prometheus using prometheusgate and that's going to allow us to say you know what, until this has been uh error free for 10 minutes, we're not going to run the promotion to dig into cloud events very quickly. To give some background on what cloud events are they're a vendor-neutral specification for defining the format of an event.

A

It's just a way of having consensus about what an event format, what the event envelope should look like, and uh it's not going to work, of course for everything, but for most things having some sort of consensus couldn't work and that and that really eliminates a lot of headache in the early stages of project, especially but most importantly, and the thing that I like best about cloud events is that they provide an sdk for python, go rust, ruby, java, c-sharp php.

A

So if you're working in a across teams or inside one team or across languages uh a lot of times, it's uh very nice to have those sdks around the spec for cloud events is pretty simple. There's four required fields. The id field is required, as is a source where the source of the event is also the spec version.

A

In this case 1.0 and the type uh the type is an important field, because it kind of works like uh namespace a lot of times, or um you know a way of kind of really thinking about the hierarchy of your event structure, as you name them, and the content type is an optional field.

A

As is schema, the extensions are optional, as is subject, and the payload, of course is where we're going to be, including our payload, which in our case will be serialized json, um a serialized json payload, which we will see here on the right here. That is, the json payload. We are sending from our local notebook up to our build context that just has a service name and a service version, provided that the rest of the magic can happen and on the left there are the fields we just discussed.

A

The content type is json, I am set to the source and the type is build.canary.release.encoffee.

A

So bentomelo you mentioned is going to be a big part of this workflow we're envisioning here. Bentoml makes it easy to build api endpoints and to save ml models, and it supports most major frameworks, which really is great because it supports tensorflow and keras scikit, which we're using onyx, pytorch, xgboost, h2o and fast ai. So again, a lot of overlap here.

A

A lot of uh support for lots of different libraries means lots of interesting integration opportunities and really it can kind of also in a way store, is a common interface to store in different to store and retrieve different model formats. So across teams, if someone is using uh keras and another team is using scikit and they're interacting, I mean there isn't any reason that this same uh technique can be used.

A

uh Models in bento ml terminology are packed basically loaded into the service that you define.

A

So if you define a new service that you want to run an api for, you load the model, the train model, so to speak into that service and that will be allowed, and that will allow you to save that service revision to a back end where it will be versioned and kept these models. Then, once they're saved to that common backend, they can be served or retrieved anywhere via the cli and the retrieve part is really interesting because it gives you an entire build context with it.

A

That's where tecn comes in the pipelines for tecton provide resources for declaring cicd pipelines.

A

The pipelines themselves are defined as sets of tasks to be executed in some defined order and tasks themselves are just pod runs and they have multiple container steps declared to accomplish the task. So you could have a task with six steps and a pipeline with three tasks.

A

The input and output between steps is generally provided via a physical volume.

A

Triggers is a project I had the privilege of creating with a few wonderful people last year. uh It's literally just at this point an http event, listener that accepts events and has some very powerful capability, in that it allows you to extract information from those events and populate a templated, a templated kubernetes resource as a result, and these resources are limited to just tecton resources.

A

So you could template out a pipeline run, for example, which is what we're doing that opens some doors to some really cool integrations, this being one of them. I think, and task catalog is another thing that we're taking advantage of a lot of the stuff.

A

We're doing here has already been contributed to the task catalog and more will, but it's just a catalog of resources designed to be reused, which is a beautiful thing when people stop having to rewrite the same thing over and over and over again, and we have a generic resource for it um and they're just yeah little composable pieces of code highly useful, highly generic, and I find that very unix-like.

A

Prometheus I mentioned as a component in our system because we need a way of automatically saying okay, the canary has done well for the last 10 minutes. Let's promote it. Prometheus is just a systems and service monitoring system, extremely popular, as I'm sure most folks in this room have had some exposure to prometheus, fundamentally stores data as time series and it collects or scrapes any configured targets and there's a number of ways to achieve being scraped for our purposes really interesting.

A

That exposes an http api because that's what we're going to be using to make range queries from our prometheus gate via the provided client libraries, so that we could pause pipelines when we need to wait for conditions to be true, such as when we want to wait to promote our canary and kubeflow serving. Lastly, super interesting project: it builds a top k native, and so we get that same scale to zero capability, uh but the infant services that kf serving builds even further and introduces the defaulting canary model right into the infra service.

A

So that's what we're going to be deploying is a canary with both a default and a canary in our infra service, and then the kf serving will expose the predictor, and that means that the deployed model server is going to be called. The traffic is going to be routed to it. If it's not scaled up, it's going to be scaled up. It's going to have its predict function called which we defined in bento ml. Remember, and it's going to return the prediction.

A

So, let's check out our demo environment we're going to have some aggregated pod logs in one console and we're gonna watch the ml namespace in another, just because that's where are gonna, we're gonna populate our machine learning, uh inference services.

A

So, let's pop over to our notebook here so in our notebook here. What we're doing this is setting the backend service for bento ml that I had mentioned so we're going to set that to our cluster so that we can uh save our models to it. We're going to write our model. This service you can see here. This is how bento ml determines that we're going to be auto resolving dependencies and also that we're going to be serving a scikit learn model.

A

So now we can have our bento ml service defined this little bit. Here is actually the scikit-learn demo data set we're going to be loading. The iris flower data set and we're going to be using our recently defined iris classifier as well.

A

So let's build and fit our model with the example. Data set we're going to be using the svm.svc model and now, let's pack, that model into our bento bento ml service and, lastly, we'll save that model to our bento ml backend.

A

While that's working that will take a minute because it's actually saving the model there, we can see that bento ml also provides a pretty interesting web unif. Web interface shows us all sorts of interesting metadata about each service that we've created and stored.

A

And over here in this tab, we can see that the service is still chugging along. It's pushing up the pickled model, each ml format it supports will save to whatever it is natively supposed to save, to and bent oml just kind of handles the pickling and unpickling.

A

We can see that it was saved. We got a new, unique identifier here and we can see that our artifacts are also saved to our s3 compatible backend. So, let's see what that looks like on the cli, let's use this bento mlcli to list our results. We can see we have a new uh release or excuse me, a new service.

A

That's been defined as of 23 seconds ago, and if we, if we define that- and we want to retrieve that service that we just defined this is interesting because it's not just going to retrieve just the model in the serialized format.

A

As we can see, we've checked it out to temp c. We get an entire build context from bento generated for free, which is really useful and that's what we're going to be using in our tektin cluster. So, let's create a cloud event. This is using the python sdk.

A

We can see that our payload here is service, name and service version. These are just the versions that I was just talking about. This is all we're going to have to tell our cluster, because the rest of it can be inferred.

A

So we're just saying we want to build an image for our iris classifier model that we just created, and we want this version. The one we just built to be built into an image and we're also specifying our build.canary.rt coffee event. That's going to signify to our trigger how we're going to want to handle this. So let's go ahead and create that event.

A

Lastly, let's again use the cloud events sdk to send our cloud event to this is our tekken triggers event listener, which will parse that json serialize payload and as fast as we can switch over? It's still not going to be fast enough, because stuff's already going to be happening in our logs on our remote cluster.

A

We can see here that the bento ml service, in the step in the tecton step, has already run and retrieved our iris classifier version that we just built and now the canico build step is going and this build step is going to do a whole bunch of things. But while this takes a few minutes, let's go back and talk about. What's going on here,.

A

We can see in this diagram if you start at the dashed developer local place. If you were to go down, that's what's going on now we started the pipeline event via cloud event.

A

They deployed the canary deploy pipeline and we go down this left side here. Bento ml retrieved, the model artifact and the remote cluster, with the entire build cluster, build context in tow pre-populated into that directory.

A

Finally, or next the caniko task build, all it has to do is take from there because it already has a defined docker build contacts directory. It just will say: hey I'm going to go and run this and build the image and save it to our docker repository.

A

That's what's actually going to take the longest here, it's probably going to take like three or four minutes, so I'm going to talk slower, but once that image is built, as you can see, the next task here is for kubeflow serving service to create the to create a canary and that canary is going to get 10 percent of the traffic and 90 of the traffic is going to continue going to our previously stable image.

A

Once that's been create done, the final step is to send a cloud event, so our pipeline sends a cloud event at the end to the tech and trigger again, which will then start the promotion. So the canary promotion process starts immediately. It doesn't happen immediately, but it starts waiting for the correct conditions to promote immediately, and that is this pipeline here once the conditions for prometheus have been met.

A

In other words, our range query says something like, after five minutes of there, not being any endpoints that have crashed, allow the canary to be promoted and then 100 of the traffic will go to our canary and that's just a way of of kind of catching any obvious silly problems that you might otherwise push out and cause downtime.

A

I'm gonna have to speed this up a little bit so this this event listener here. The red parts are what I was mentioning earlier. You can use these event listener specs to define the interceptors, so you will match the headers in this case. We know that the ce type will contain our event type. So with cloud events, it's really easy to standardize on using these types to do predictable things all of the pipeline tasks. We will run I'll, try and talk through quickly.

A

First, we're going to use bento ml to retrieve the build context. Next, we use canico executor to build the context into an image. Kaneko is a a docker image, build utility, but it does not require docker daemon to build, which makes it ideal in a ci format.

A

Then we're going to use the kf serving carry creator canary creator task and we'll create a canary deployment for our new image. Like we said, it'll get 10 of traffic and then our prometheus gate will wait five minutes without seeing any pod failure metrics once it has. It will allow the canary promoter to run so what we can do now is. We can take a second and check out our canary deployment once that's done.

A

We can query our canary or if it's already promoted unlikely our default endpoint and uh we can watch it pass from 10 to 100 traffic. So it looks like here we we are looking at our build process. Still the canico process. You can see it's doing a whole lot of pip installing python modules, it's just building away and right now, what it's doing is pushing up. All of the um is pushing up our docker image layers up to my docker repository, but let's, let's get all the pods in our build namespace.

A

You can see here the build tasks themselves happen as pods and what happens inside them happens as containers as steps. So you can see our bento ml test completed very quickly, but our caniko build is, is taking much longer, not only because it's running um a much more process and has intensive workload building an image. But it's also, I think it's doing like eight different steps. So there's a lot going on and we can see that it's finished now.

A

So if we were to go back and watch our ml.

A

Namespace we can see a lot has happened. We have two pods initializing up here and even though we're set to auto scale down to zero it'll, initially come up and uh boot, these pods right off the bat and then, if they don't get any traffic for a while to go to sleep, but we have, you know noticed we have two pods a canary and a default.

A

We have two sets of services for canary in default. We have two routes for canary and default, so we're really just running two deployments and having the traffic routed 10 to one 90 to the other.

A

So what we can do is we can take our route here, which is model dash, imp dash, predictor dash canary and then an xip.io here.

A

And we can make a query on our uh ml endpoint that we just created, so we will set if we set service equal to this, and we threw a.

A

A

A

And we want our input path as well to be set to um something usable, so we will create that input right now,.

A

And then we should be able to make a prediction and we can see that we get our results here, so we were able to query our canary and we're able to get a result back and let's see what's going on in our logs. So here you can see the prometheus query is uh underway and we're just awaiting promotion of the canary.

A

At this point, you can see that the value here is four, so it's going to take a little bit for this namespace to settle and once it has, it will run through a promotion step and we will see the canary become promoted, and that is the entire demo.

A

Thank you for your patience as I work through uh this new format, and uh I find this a very interesting topic. I'd love to talk more about this with you, uh so uh please do engage about it and have a great day.

A

Yeah, so thank you so much for listening to uh to this session. If there's any questions, I'd love to answer them about anything to do with cloud events or bento or anything else we discussed here tecton.

A

uh Otherwise, please do stick around. There will be a sponsored video right after this and um I'll wait for a little bit for some questions better than that. Thank you so much uh for the opportunity to speak.

A

Right looks like we have no questions so again. Thank you so much and take care cheers.