Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 11 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building Distributed TensorFlow Using Both GPU and CPU on Kubernetes [I] - Zeyu Zheng

Description

Building Distributed TensorFlow Using Both GPU and CPU on Kubernetes [I] - Zeyu Zheng & Huizhi Zhao, Caicloud

Big Data and Machine Learning have become extremely hot topics in recent years. Google has announced its AI-centric strategy and released the deep learning toolkit TensorFlow. TensorFlow soon became the most popular open source toolkit for deep learning applications. However, it may take years to train large deep learning models on a single machine without GPU. In order to accelerate the training process, we build a distributed TensorFlow system on Kubernetes which support both CPUs and GPUs.

In this presentation, I’d like to share our experiences about how to build this distributed TensorFlow system on Kubernetes. First, I'll briefly introduce TensorFlow and how TensorFlow supports training model distributedly. However, the original distribution mechanism lacks lots of components such as scheduling, monitoring, life cycle managing and etc. to make it suitable for production usage.

In the rest of the presentation, I'll focus on how to leverage Kubernetes to solve those problem. The solution involves three components. First, I'll introduce how to schedule TensorFlow jobs in a cluster with both CPUs and GPUs. Then I'll share our experience in managing the life cycle of a distributed TensorFlow job. Finally, I'll state our efforts in lowering the bar for using distributed TensorFlow

About Huizhi Zhao
Software Engineer, Caicloud

About Zeyu Zheng
Zeyu is chief data scientist and co-founder at Caicloud which provides Cloud and Big Data related services. He leads the efforts to build reliable and scalable data analysis and machine learning platforms like Hadoop, Spark and TensorFlow on Kubernetes. His team has developed Machine Learning applications like image classification, time series prediction, which has helped well-known Chinese enterprises to utilize machine learning based on Kubernetes in production. Before I co-founded Caicloud, Zeyu worked for Google Shopping for almost three years. He proposed and leads the efforts in building structured product cluster data which is an essential part to trigger Knowledge Card product ads on google.com. Zeyu obtained his Master’s degree at School of Computer Science at Carnegie Mellon University (CMU). He was named Siebel Scholar, which only awards 93 graduate students from world’s leading graduate schools based on outstanding academic performance and leadership. During his internship at Microsoft Research Asia, he published several papers and delivered academic speeches at top Data Mining conferences like SIGIR and ICDM.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Thank everybody for coming and and Bloomberg and chief data scientist at tax out and today, and my colleague and I, will give a talk about how to run tensorflow uncleanliness with GPU enabled and we'll also introduce you guys about. You know the experience we have turned from rounding, the distributor tensorflow and also, and about some something about tensor flows.

A

So here's the agenda for the game. First of all, I, would like to introduce you about the you know, some basic ideas about tensor flow. What is tensor flow and how it's related to deep learning and artificial intelligence kind of thing and then never like to introduce about you know how to start distributed in the flow and mostly around mental. You know machines and then I would like to introduce a little bit about how to start.

A

You know distributed tender flow with Cornelius and then I would like to introduce my colleague which told you to introduce about how to you know, then enable GPU on coronaries, who was the author of the you know: github pull request flowing neighboring GPU and after that I would like to hear about how to you know, enable the tender flow with you know, to puing able junk religious and also well, like you know, what's the benefit of it and how do I do that?

A

hmm And finally, I would like to introduce you about product with building and we are going to announce the first version of the public cloud about you know, distributed tensorflow and it's named as tangible asset service.

A

So first of all, I would like to introduce a little bit about deep learning and, as I think most of you should be heard about. You know the deep nurnies and stuff and I think the first time that most public heard about steep learning and something like artificial intelligence starting from the ethical project so which is a goal player and to win who win the leases.

A

I, don't know how to translate his name, but it's actually a world champion go player in Korean, so at and after that half ago also wins like 60 top players in goal game, and that makes public most aware of you know new technologies coming and which is named names, deep learning. So, except from the you know, some fancy game, game and Tiffany has already been applied to a lot of stuff.

A

Like an aside, I show the Google car project riches, you know starts from mccurdy and 2009 something a cat and it has already been tested for, like two hundred thousand miles kind of so Richards from very long distance, actually and doesn't have much problem of next close crash that nobody injured genes. Actually, nobody died in an accident so which is a you know great achieve, but Mr Chief cannot be.

A

You know it cannot be achieved, results, a new technology in active learning so and this line shoes trained in Google, search about the term of deterring, and you can see from the graph that starting from 2012 is going exponentially, and so it's just more attention from year to year, and you can see it's from the turning point is 2012 and starting from that up to 2016 becomes. You know most of well-known actually in terms of the technology.

A

So what the turning points come from 2012 and what happened in 2012, so I will just give who you are very simple extent, and you know the most no example, and which is you know, the most most of deep learning applications has already being started.

A

So the this is a picture from an image net which is image recognition, data set organized by the chief data scientist at Google, which is a previous professor at Stanford University, whose name is Lisa T, and so she organized she just gathered a lot like you know, almost say: 10 Wow, let's see, 1 be 10 billion kind of more than 10 billion pictures from the different software sense label, dreams from Mechanical Turk and having some picture likely, organizing avail, you know, will organize the form in the word net.

A

So mostly it's trying to capture the idea about. You know what you know what's showing on a picture. Like you know, in this picture we saw some shears and with a dog and the person in the picture. So the purpose for the data set is to you know. That's a machine learns how to recognize the picture and which seems a very easy.

A

You know task well present, but it's actually a very different task, for you know computer, because you know University of you know you know, computer, you just saw numbers right, you don't actually have advanced and have the version. So it's hard to attitude that computer to learn about what is what so, then, what happens in 2012 is that deep learning technology totally change the story in image, your calculation area, so in this process cute accuracy of the algorithms before the dip during and after different, you can see there's a huge jump in the error rate.

A

So it's before learning has been implied. The error rate is about like 25 percent, and after that it you know, starting from sixteen and down to four now it's about just about to represent and it's much better than what we doing a human-like, because you know we are one in that data that we have about 1000 different categories and some people, like you know we have very similar, and even people cannot recognize it. You know 100% sure.

A

So that's what happened because of the you know, and technology of the deep learning and image recognition is just the beginning of the you know: booming areas like if during this journey has already been applied to like text and translation and search recommendation. All those areas has has been changed by deep learning. So it's a very you know server, so so this technology is fascinating.

A

So how can we leverage the power of deep learning to a lot you know daily product or to our you know real real hmm everyday use, so we first look for we need to have a tool like you know. The stock tip technology is very great, but without a tool you cannot apply the technology tool to a real system right. So there are lots of different rules. So in this slide, I just give you some idea about. You know most of famers open source of deep learning tool.

A

Kids, so it just lists a few as they are more, but from the graph we can see that the tender flow attract most of the attention in the community right. So the green bar shows that has a flow and there number number of folks and number of issues and pull requests. So the first two graphs use that you know how popular the toolkit is, because more folks, more stars means that more attention from the public, but more issues and pull requests means more people are actually using it.

A

So this is one reason why we we would like to introduce. You know tensorflow as a toolkit to implement deep learning, algorithms and another reason for that is you know it. It has been already applied used by a lot of big names like Google, Hoover and treater deepmind defenders, actually that part of Google right now and also some you know, companies in China. So that's the reason.

A

Why would book mind you know tensorflow as a tool, kids to implement deep learning, algorithms, then the rest of talk would be focused on you know: how can we better rather tool kids and how can we better leverage computing power of the deep learning? As you know, deep learning is not a new name right.

A

It has been alone for and it has been proposed for like 60 years already, but why now we can you know the the neural network or deep learning technologies comes again most of these because of the you know, big data and computation energy. So we have.

B

A

Computation resource and we have more data which makes a you know, deep learning, better I wisdoms and some traditional machine learning, algorithms, so the cement. We need to actually leverage more computation results. And how do we do that- and this is the focus of the rest of my talk so and display intensive so on?

A

One server is actually very easy and you just need to pipe installed, handle flow and it will install the latest version of tensorflow, and you can see that you know after you install GPU on your own machine, it's very easy for tender, flotool average power of cpu. You just need to install a GPU version of the tensorflow, and it's all there.

A

You can just install it on your own laptop or your arm server and it's there, but one tenth of you know the single I mean the telephone that one server just can average the resourcing a server hand, no matter how strong the server is is limited in computational resource right, and here is the result.

A

So if you'd like to train, you know the planning algorithm on the image metadata that I just mentioned, it will take about half a year to achieve the accuracy about 68, so even to achieve the similar accuracy, research, traditional, you know, machine learning, algorithm it takes at a half a year. So that means you need to leverage more computational resource in order to you know, have a better result or have a better results in a reasonable amount of time.

A

So that's why we would like to introduce distributed tensor flow, so tender flow, actual support distribution starting from zero point, eight version which released back late April last year, and the high-level architecture looks like this. So basically it has some terminal servers and some some workers right so but family service and just starts parameter in the you know. You know if learning algorithm and the worker does the calculation work, so most heavy calculation stuff happens in a worker and the parameter server just holds parameters. So it's just done.

A

Take a lot of computation or results and see a lot of you know: network communication between workers and parameter server. You can see each worker communicates to all terminal servers, so it's a kind of fully connected graph. So let me have some problem in the networking. Bandwidth and I will talk about that later and so distributed hinder flow also can leverage the GPU. So here is the architecture about. You know how to leverage the GPUs, because terminal server doesn't do the heavy calculation work.

A

So we don't actually need the GPU to attach to this parameter server, but for each of the worker we can attach one or multiple GPUs for each worker, and that's you know, from from the from pure class the point of view with: let's distribute it has a flow, and it's all supported by the original has a flow. Then what we like to you know introduce how to run tangible and proven areas, what the kubernetes do, and so that's what have were introduced.

A

So, first, the you know, tens of distributed tensorflow don't have a management enough stuff. So basically we this is always needed. You know what we need to robotics robustly wrong distributed sensor for, like you know you, if you run the distributed sensor for you in your, are machine or your own server, then you need to your log onto every server and start a job, and also you need to monitoring the different servers on your arm and also some measurements kind of things. Also. You know it's not supported by original tensorflow.

A

So that's why we need :. It is to host the tensorflow. So how do we do that without GPU? We can actually leverage. You know different different. You know strategies for for running a distributed sensor flow from here. You can see that we're on the terminal server as a Canary's deployment and random walks are a support. So basically it's so it has different different. You know strategies to start. What do we do that so for the premiere server?

A

Actually, it's it's quite it's not so complicated to start and also it's ever ever running job intensive flow. So we try to use deployment to start the tether flow before for the worker. It's actually a bad job, and so we we use the best job to start turn the flow, Walker and Israel. You know automatically creates updates and the iteration ends for the training part. And so then we can distribute difference. You know we can. We can either assign the you know, tender floorwalker to a same server or we can assign to different servers.

A

So the observation is lat if you assigned to so tender Fluke actual average, all the resource singer, your machine. So if you're running on a single machine almost and but so in that case, if you remotely pour water on same same machine, the speed, you know the acceleration wait, we are not be so quick, so you need to assign the worker into in the different servers and also you know for that, so a PS and feminine server and worker you you're, better assigned.

A

You know PS close to two workers, because there are communication between workers and premier servers. So that's how we can run tender flow, distributed, Center floor plan. It is but one if we need to you know, leverage GPU power. So we need to actually make sure that the coordinated cluster can can. You know, manage the GPU resource and then the this part I would like to introduce my colleague Java G, to introduce how to enable GPU octo analyst.

B

Hello, ok I would like to introduce that how I enable GPU and grenade in communities and where we need a GPU I, think GPU can benefit lots of applications, especially photons of law, and also it can benefit lots of other applications. So GPU is very important and to enable that I think I start to enable the GPU I was working in AP and the first PR I mean it's also from there to enable GPU I think that we need to have different levels like.

B

First of all, we need to make the sequence of whole the GPU. Also we need to the container container manager like docker or rocket principle. The GPO understand we will think about the ultra environment like grenades or metals or other things, and since we are using the kubernetes and stalkers of outlet introduced, hallway enable the GPU in poker and the coronaries.

B

First of all, we need to see that 30 was matching in docker talker I used the GPU as the device. As you said, our device is already been supported and the third we need to give the GPU discovery capability in relax because the we may have a container all treatment environment. We may have like thousand servers and we do know where server have GPUs and we need to let the crew, like our trip you by yourself.

B

Now we configure them one by one server and then we need to have the assignment and free GPUs in relax. Once we have continue around run our machines that we need to assign GPO to a container and the when the container is tired or killed, we need to free the GPU so that, if you can be used by other containers and the that's, why is Scalia GTO resource in Hoover scheduler? The crude carrier need to know how many GPUs are.

B

Each machine also eaten need to can have the capability to assign multiple GPUs to one container. Okay, let's go through them one by one.

B

A24 Jolly Ranchers talker is already conceived of the device command the devel come oncome, treats the really a talker as device. The u.s. is one of the sequel had name space city west. Have we already have the CPU memory and like blog I'll device or networking, but for other devices, and we need to treat them as device and if you want to as the GPUs through container, we need to add integrity to container, not just like the networking like maybe or the CPUs, a twenty percent of the city of we can Cassandra.

B

Can anybody can do that for GPU in this case, I attached a to can two GPUs to one container at this desk command to use the dead edge command, and we must ensure that the media control and media uem also be attached to the container, and otherwise it cannot be used.

B

Once we run this a command, we can check the radio device inside the container like to check well lighter whether the video, zero or nvidia one it's already under the dial, tell folder. Also we can use the dagger inspect command to check the device matching. You can see that you have the path and host and the past in container also the permissions.

B

This is a hallway assign, can CPUs to container uncover.

B

After that, we need to make the give the kugel I'll have the discovery capability of diffuse out. We can see, as in this theater, that we need to attach the video CTL and media uem to the container wherever GPU device you want to attach the container and only the immediate abuse. So far we only use the radiative use and eyes. We need to ensure that the coda has already been installed and the Video Grabber is also installed as well, and we can check the use like use the nvidia SMI the command.

B

We can check whether it's a installer correctly on this machine. We can have. We have like four GPUs and it will show like this. We have immediate zero to a media for and also we have the media attention immediately. mmm Also, we need to decide which kind of information that we need, like the coded version or the just the GPU pass, or some like family names, because for tensorflow is, if we use different kind of GPUs, it may have a lower performance than the same time, so they need to so it's better.

B

We can have the family name. Also. Trevor name is very important, but so far it's only suppose the GPU as device like we have the radiator or in either one. We don't have other information because it Gardens with other informations. We need to use the Aradia management library, it's called only ml, but it's so far. It does about that yet and we are working on that.

B

And next one is how to set and free GPUs in cobra light in the future. We can see that we assigned to two GPUs for one container and the 30 pills 140 fields for one container Colet need to management. The BPO information is Beverly like the GTO. Pass is very important and for other informations, if we can support it in the future, it also will be in the core alight, but actually incorrect as a temporary version. For now we will move that to the city wiser. It will be better.

B

We can see the scheduled GPO resource in the Coover scheduler, or we triggered immediate GB result as the CPUs and memories we put. It is here so once you read like a Yama file that either contingent the GPU is a resource that you need, the will can allocate like multiple GPUs for one container, but so what? If only dedicated GPUs, because that way you sleep use like, as in the device group, not as employment in the kernel level, like the CPU memory, plot device or network.

B

B

As it is, but what we should do next for the first line is that we need to support the CRI and so far it's only support occur because we use you know demise scenarios and we only you saw her, but the CRI is already in the 1.6 and we will like the NVIDIA GPU crucifer, the CRI. In the next version and like I said we ml is very important because we want the family name and the driver version, and maybe the sloughed information is also needed.

B

But if we want that, we need to insert we need to load every ml library that, first of all, we need to support a GPO discovery and GPU assignment. Also the like a multiple GPU in one hand and is what we need right now. We need to do that once there I was there and that's. Why is a GPO 12-volume support, because the GPU driver have a comparable issues if we use a higher version and we need to make different volume for the different container, so we need to like.

B

We need to know the family name and the cooler version. So we can judge the amount of volume automatically as we do need to read it in the llamo file. That will be quickly. We have the GPO travel amounts, but we need to enable the vml library first and the last one is nico support this about the collaborative communication, like maybe our emotions that we have lots of GPUs and GPUs can either can communicate like a very spot and have lots of dates communicate.

B

It will be the bottleneck when we rendered in the flow and if we suppose, a nico I think that we can resolve that problems. But we need to do those things one by one and you have a lot of things to do, and we have a list for that and I think it will is very welcome that if you can join us to enable the GPUs, ok.

A

Yeah, so actually that that's a part of the you know problems every encounter about how to no Butte and sustainable tender flow cluster on top of Cronus. So the one of the most important problems how to actually you know enable the GPUs in the coroner's cluster after we enable the you know, GPU inside of coronaries cluster. The cleanest clutter knows that okay, some of the server has to be um doesn't and after we, you know, schedule the Tendo flow Walker.

A

It will actually search for the no GPU which search for server, which can actually you know, actually have a GPU supports. So, but actually we can assign the workers to the server which have TP resource and the sign the you know, chip and having a server to know to a worker that may not have a GPU and device. So the multiple ways that we can we can distribute that I mean distribute GPU resource. One is that we can actually use as many as GPUs. It has a server.

A

You know talker and in that way it's mod fencer, and sometimes you may achieve better acceleration rate. But the problem is that you know the user needs to. You know be aware of the number of GPUs it has and it need to admit you know manually set which operation should be. You know allocates on which GPU in each worker, if you, if different worker, have different, you know number of GPUs, then it will be a problem to you know to start cluster, and so in that case we choose not to support much much abuse.

A

You know worker at least I'm, but we will work on that. You know you know you know future for now our solutions that for each worker we all need support at most one GPU, and then we can. You know if you have multiple GPUs in your. You know server that case you stopped in the left left side and you can actually allocate more than one worker in to that server. So that's a solution in our system right now and curing is a kind of reboot distributed tensorflow.

A

We actually encountered some some problems and there are some solutions. They are some of the discussion. I would say so. The first first lines, the network is actually the worst bottleneck and so in the tender flow official you know website allowance today will almost accelerates linearly with the number of GPUs it use, but in all experiments we find that the mini narratives is not so easy to achieve the most.

A

The hardest problem is the you know network why we sell ads, because two different tasks actually needs the network once the planet server needs the network to communicate, which workers to translate I mean and to to commute, communicate on the new parameter server I'm, not having server parameters in the model and the nollans worker needs to consume the network to load data. So both of the you know, furniture and worker needs a high amount of network resource.

A

So in that case it will you know the network is easily get you know, consumed up and they've become a bottleneck of the acceleration, so some some of the strategies that we try. This first of all, you can actually, you know, have a you know better network network strategy. That's that's! That's you. You know that is what you can choose always and besides the pieces that we try to actually adjust a number of time. You know server, you would give in the heavy flow cluster and also our experiment.

A

We saw that if you increase a number of you know, batches in the tender flow training job. It also is the problem of you know getting too much communication between terminal worker, Prime, Minister and workers so, and they are definitely some more strategy that you can and try like you know. You obviously have some configuration that you can try different.

A

You know, strategies of the like you know, to better, have a better communication reach, but that we have a trial episode that we have low Detroit and some of them worked to you know in our experiment. We saw that six GPUs can achieve about 3.5 times duration than one GPU. So and another scenario. We also try this- that we try different GPUs like 1080 and Katie.

A

They are different, very different kind of GPUs when K it is actually four server and it's it's running very smoothly and also it have lower risk of burning out and but the publish that if you have Nokia about the speed, then thing it is. You know what is what you need? Actually, it is faster than the speed of KT and its path, shiver, soul.

A

It's some some problems and some you know, experience weaker and some from a lot hmm previous experience and but the polling I would like to introduce about our product that is ten to flow as a service. So what what slack? That's what we do in the system? So basically we have so so mostly it's in Chinese right now, but it will get translated into English in something around like maybe late, May or early June, something attached. So you can just look at the interface mostly it's just.

A

You can start start to tensorflow training and we have a higher level API of tensorflow, which looks like tf9, but it's really different and also we handles distribution automatically and we handles all the monitoring. All the you know, resource management and stuff like that in the your system. So here is a very simple. You know: different I mean the the function function and compare meant between the original distributed has a flow, and our product has a flow as a service.

A

So, in sum, is that if you know the hydro system, which composed of HDFS filesystem and young scheduling system and the MapReduce computational framework, so what had to could be so popular is because it's all self-contained and you can actually deploy a tensor flow result lot of tricks. You know the system stuff, you can just leverage it and you can put your data on the HDFS and you can use a MapReduce commutation framework to do your calculation, but tens of flow only support.

A

You know calculation framework, so it supports distributed distribution and it has the mode of you know: distribution, that's with the leak is that it leaks of the you know, resource management like we need to have a coronary cluster to allocate you know the jobs and monitoring the resource it used. Pat has a flow cluster and also it needs a you know. Like what will happen if it, you know, some ever happened in a training process.

A

You need to restart the training tuning process and also you need to see whether it's because of your algorithm or your data or you, you know you plan form. So in that case our our system will notify you. You know whether is because of you, you add, with the noise because of the platform. So that's what we did and also for higher level abstraction that you know for deep learning model you will not have to just trim wise. You will try different and layers. You will try to.

A

Even you know, structure of the you know a newer network. You will try different hyper parameter. Like you know, number of nodes, like your learning rate number of patches I, mean number of training staff stuff like that. So mostly you will. You would know tree much people and you will run multiple Center for distributed, has a flow jobs so yeah, so lots of difference between the original type of flow and our technical as a service. So there are some some interface because of the time is limited. I.

A

Just want to just very briefly show the interface and stuff like that, so we have a storage which must look like you know: local local storage and also we with monitors the training process and resource usage stuff like that. So and there are some some internal resource management, so we have a resource pool which every job runs on resource pool and we have a curing strategy that you know after you use all the results in a pool. You will be queued up in the you know, waiting queue and you won't get a video every time.

A

You start you're trying to start a new channel flow job, but you don't have enough resource in the queue. So- and here is the you know our company architecture we have. We have, we support different cloud providers and we have the infrastructure. We provide tens of eliezer service and all we also provide AI solutions. So that's it and we have some Facebook and Twitter links, but if have WeChat never be better because they have more resource, then that's it. Thank you. So much.