Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 7 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Scaling Deep Learning Models in Production Using Kubernetes - Sahil Dua

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Keynote: Scaling Deep Learning Models in Production Using Kubernetes - Sahil Dua, Software Developer, Booking.com

While there are a lot of machine learning frameworks and libraries available, putting the models in production at large scale is still a challenge. Sahil would like to talk about how they took on the challenge of deploying deep learning models in production: how they chose their tools and developed their internal deep learning infrastructure using Kubernetes. He will cover how they do model training in Docker containers, distributed TensorFlow training in a cluster of containers, automated re-training of models and finally, the deployment of models to serve predictions. At the large scale which they operate on, nothing comes easy. He will also talk about how they optimize their model predictions infrastructure for latency or throughput depending on the use case.

"About Sahil
Sahil is a software developer at Booking.com. He has been involved in leveraging container infrastructure to help Booking.com’s internal teams in taking advantage of deep learning techniques at scale. An open source software enthusiast, Sahil is a core contributor and community leader for DuckDuckGo's open source organization. Besides that he is one of the contributors to the Git project, pandas - open source data analysis library, Linguist project by GitHub and Go-GitHub project by Google. Sahil has been actively speaking at conferences in last one year including FOSDEM, EuroPython and SIGNAL 2017 to name some of them."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hello everyone so, as Kelsey already introduced me, my name is Sahil dua and today I'm going to talk about how we build a production pipeline to deploy our deep learning models in production. Of course, using humanities, I hope, I covered all the buzzwords.

A

Ok, so I hope. At the end of this talk, you will feel more confident in putting your models to production serving traffic serving predictions at whatever scale you work on. So let's start before I start. Let me give a bit of intro about myself. I work at booking as a software developer, previously I've been involved in building the deep learning infrastructure there to support our deep learning models, and that's what I'm going to be talking about today.

A

Besides that I am a big open. Source fan, I contribute to a bunch of different projects, and my most recent biggest contribution has been to the git project itself, which I'm really excited about, and besides that I have a frequent tech. Speaker, I normally talk about topics like a be testing data analysis, deep learning, containers and a bunch of stuff around that.

A

So, let's see what I'm going to cover today, I will start with mentioning some of the applications of deep learning, some of the things why we felt that this is an important problem to solve, then we'll see how we train our models and then we'll see how we use those training models to serve predictions, how we deploy them.

A

So, let's start, let's start with the deep learning applications and before I go there. It's really important to understand what scale I'm talking about here. So at booking.com we have more than 1.5 million room nights, booked every 24 hours, and these bookings come from more than 1.4 million properties that we have 220 countries.

A

So let me make one thing clear: I'm not here to brag about these numbers here. The point I'm making here is that at this huge scale we have access to a large amount of data about our users and we can use that to improve their experience on our product.

A

So, let's see how we do that. The first application of deep learning that we saw at booking was image tagging, trying to look at a image and identifying what objects are. There is a really important problem for us, because then we can use that information to have extra meta information about properties that we have on booking.com.

A

Another problem that we are solving is translations. We are trying to build models to translate the text that we have from one language to another without any human intervention, and this is a data. That's really specific to our travel domain and it's not like publicly available translations can't do that, but we're trying to be really domain-specific here and trying to translate our own stuff, which is later to travel, and we bid quite a lot on advertisements on various search engines, and we want to be making sure that we spend the right amount of money.

A

We built the right amount of money on the right kind of keywords so that we have maximum return on investment. So we have our models in place which, using which we predict how much we should be bidding on what kind of keywords and when we should be doing that, and we have a bunch of other applications of deep learning where we deploy our models, deploy our resources to solve some complicated problems.

A

Now let me talk about one application in detail, so the image tagging. What is the problem here? So this is just one question that we want to answer. The question is: what do you see in this image now? This is a really simple problem. If we ask any one of you, you can easily see you they're in this image. You can identify the objects, but it's not really easy for a machine to be able to tell what's there in this image and what makes this problem even harder. Is that the context matters a lot?

A

Let's take an example, so if we pass this image through some publicly available networks which detect the objects in the images, these are the results that we get. It tells us that we have oceanfront nature beach house, it's a building, okay, but do we really care about that? These are not the things that we are concerned about at booking.com.

A

However, these are the things that we care about. We care about if there is a sea view from this room from this property we care about, if there is a balcony Terrace associated with this property. If this photo has a bed object, any object like so far chair or if this is a picture of inside view of a room and here's one more interesting thing about this problem.

A

There's going to be hierarchy of the tags that we are talking about, these labels there is going to be hierarchy of these labels here, for example, if you see that there isn't there is a bed in this image, you can be sure that this is going to be an inside view of the room.

A

Unless you are in such a room where there is no room, just a bed.

A

So yeah once we can identify the objects once we know. What's there in an image, we can map this information back to the properties that we have and use this information to help our customers in finding the relevant properties that they are looking for easily and quickly.

A

So now that we have seen one application of deep learning, what's so special or this, what's so special about the workload that we have to run to train these models or to deploy these models. Let's talk about that like. Why do we really need something special, some different platform or a pipeline to work on these kind of problems?

A

So the first thing that's different is the deep learning workload is highly computationally intensive, which means it's possible that to run the model to get one prediction out of it, you might have to do thousands, or even sometimes millions of mathematical calculations.

A

Another thing is most of the times the algorithms that we use to train. These models are not easy to paralyze, and the third thing is this huge amount of data involved. The data goes from tens to hundreds of gigs, and sometimes the data is in terabytes as well. So we have to have some different kind of platform to run these kind of workloads to run the training to run the deployment.

A

So now, why do we choose q1? It is for that. There are a few reasons for that. The few things, cracked six, that we identified, which make your when it is a viable solution for this problem.

A

So, first of all isolation, there is a namespace isolation which human it is provides, which makes sure that the resources are not shared between the processes and those processes are not fighting not competing against each other. For those resources and second, is elasticity. Curators provides the ability to easily scale up scale down our applications and also lets us run as many containers as we want, as as far as we have the hardware- and this leads to a better utilization of the existing resources that we have. And the third thing is the flexibility that we get.

A

It provides us a easy, quick way to try out new things. Try out a library, try out a new framework in our production environment, and that leads to a faster velocity of innovation. For us.

A

Now, let's see how we do our training with kubernetes, so the first thing to note here is: we have our base images with all the machine learning frameworks that we think are going to be used, and these are normally tensorflow, torch, waffle, rabbit and a bunch of other frameworks that people are using that data scientists are using to write their models now interesting thing here is that we don't put any training code in the image, so this makes it easy for us to easily eye trait or while we are developing what the means is.

A

We don't have to go through a workflow of making a change to their training code, building an image putting it into the registry, downloading it and running a container. All we have to do at every training. Step is just run that container and tell it where the code should be fetched from. So the helps us in iterating really quickly and to run the training. We need a lot of data. We need to have access to the data. So how do we get data to these containers that we are running? What we do is there?

A

It really depends on what framework the model is using. For example, some of the frameworks know how to talk to her dupe in those cases. For example, tensorflow in case of transfer flow, we can steam the data directly from Hadoop and in the case of a framework when it doesn't know how to talk to Hadoop, we can get a persistent volume associated with the pod and get the data downloaded over there and then run the training as if the data is available with the container locally.

A

So, let's look at this process of training now from a visual point of view, so we have a training pod, which knows where the code is to be fetched from where the code is to be fetched and it takes the code. The code has some training script evaluation script and the next step now is to get the data and, as I already mentioned, can be either using directly streaming from Hadoop storage or it can be downloaded into persistent volume and then trying to use it as locally available data.

A

So once we get the data now we'll start running the training and while the training is running, we want to be able to look at the progress of it. Like look at what's going on inside that data, scientists should be able to look at the progress should be able to map back use tensor board to monitor the performance, monitor the training process.

A

So we stream the logs back to Hadoop storage and once the training is done, we want to be able to store this model somewhere so that we can use that later to deploy it to get predictions out of it. So we take the model, put it back to the Hadoop storage and we are done with the training.

A

So once we have trained our model, the next step is to take this model and deploy it so that we can serve predictions to the actual clients.

A

Now, let's see what we want here, we want to have a model we just train running somewhere in isolation and being able to serve traffic using REST API, and similarly we want to have multiple models being able to serve traffic in a similar way in their isolation, isolation, environments. At the same time, we want to scale these models scale, the applications that are running these models independently.

A

So let's see how we do that, so the first thing we have here is: we have the stateless application code with some common code, which is required to kick the server running.

A

We containerize it, and this is really important. We don't have any model embedded in the image. So now this again helps us in two things. One is making sure that the size of the image is really small, because the models can go up to a few gigabytes sometimes. So. This helps us in making sure the images are always small and the time to run the container is really small, and the second thing is: it makes the images that we have reusable.

A

We don't have to build a new image every time we train a model. We just have to kick the container with the image that we have and make it point to a particular model that we just trained, and then we expose the rest API to get predictions out of it.

A

Now again, let's look at this. From a visual point of view, we have our serving pod, which we are running in our curators cluster. It has some basic code to just get. This thing started to be able to get the model and do all the stuff, and now we know that we exported the model to Hadoop storage. We want to carry it back now, so that we can run the training.

A

We get the model back and we load it into the memory.

A

Once we have the model loaded into the memory, we want to expose that as a REST API to the client, so that clients can send a get request with all the input features that they have, that the model requires and send the predictions back in the response now. Overall, this is how it looks like get.

A

The model from Hadoop storage loaded into memory serve predictions using a REST API to the clients and in reality this is all looks like it's not always just one part, there are multiple pods running for one application, running the same model, doing the same process that I just mentioned, and serving traffic to a large number of clients using the same mechanism that I just mentioned, and using this kind of setup, we are able to scale up or scale down different models according to their requirement, cording to the demand, according to how many clients who want to connect how many clients want those kind of recommendations, those kind of predictions.

A

Now, what do we need to do to create a deployment create a new model deployment? We create a deployment deployment object here, which is going to specify the pods configuration based on where it needs to get the data from where it needs to get the model from and how it needs to get started. Once we have the deployment running, we want to create a new service so that we can expose this these pods and wait for these parts to to pass the lightness and airiness props before they can start serving traffic.

A

So let me quickly try to summarize what we just talked about so I started with applications of deep learning the applications like image, tagging translations and ads bidding. How we solve these problems or why these problems are really important for us. Then we saw how it rained on models spawn a new training pod get the code from the git repo and to run the training stream back the logs and export the model at the end of the training back to Hadoop storage.

A

Once we have the training done once we have a trained model that we now can deploy. We spawn up a new serving pod that will take the model from Hadoop storage loaded into the memory and expose REST API.

A

That's what I that's all I had to cover today and if you want to get in touch, please get in touch with me on LinkedIn or Twitter or whatever you prefer and I will be around today. If you have any questions, thank you.