Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 6 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keynote: Building the Infrastructure that Powers the Future of AI - Vicki Cheung & Jonas Schneider

Description

Keynote: Building the Infrastructure that Powers the Future of AI - Vicki Cheung, Member of Technical Staff & Jonas Schneider, Member of Technical Staff, OpenAI

OpenAI is a non-profit research company that does cutting-edge AI research. Our mission is to build safe AI, and ensure AI's benefits are as widely and evenly distributed as possible. This means democratizing the technology and releasing our research publicly. As a result, we rely heavily on open-source software. The majority of our experiments run on our Kubernetes cluster that spans Azure, AWS, and our own data center. Kubernetes and Docker have allowed us the flexibility to experiment with various computing frameworks and topologies without paying the infrastructure cost. However, our use cases are distinctly different from the well-supported microservice use case, and we've written custom components on top of Kubernetes to optimize for our work. Some examples include our own autoscaler for batch jobs, a library to deploy distributed Tensorflow jobs, custom scripts to do GPU-scheduling and CPU-affinity, and a variety of internal tools to make Kubernetes friendly to researchers who have no experience in operations. In this talk, we will go over some of the motivations and internals of our customizations, as well as an example of how they all come to work together to accelerate research on the Universe platform.

About Vicki Cheung
Vicki was part of the founding team and leads infrastructure at OpenAI, where they run deep learning experiments with large numerical compute requirements at scale. Previously, she led engineering at TrueVault and was a founding engineer at Duolingo.

About Jonas Schneider
Jonas leads OpenAI's Robotics engineering team to build a platform for real-time control and distributed data collection. In his spare time (how?!), he builds infrastructure at OpenAI to provide high-performance compute for our research projects.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hi coupe con um I'm Vicki, and this is Jonas and we're from open, AI and so yeah we're going to talk about how kubernetes has helped us build our infrastructure that powers the future of AI and how we might have started as an unusual use case for kubernetes, but we've actually managed to make it work so far.

A

So what is open? Ai we're a non-profit, AI research company and we focus most of our efforts on both basic research and deep learning, which is an area in machine learning, as well as research in specific projects. Examples include robotics and learning to play video games so as a non-profit company, our mission is to democratize the AI technology, and because of that, we care a lot about open source software. We rely heavily on them and we also care deeply about benefiting from and contributing back to the community. So.

A

Let's talk about the infrastructure, what makes building infrastructure for a research lab running deep learning different from infrastructure at a start-up, so there are two key differences. One is the type of workload that we're running, we're mostly running large batch jobs, so the jobs can span from hours to days or even weeks, and sometimes we run a single copy. Sometimes we run tens of thousands of copies, so that puts a very different strain on the cluster and the type of jobs that we run often have very specific resource requirements.

A

So this means running on special hardware, that's configured for doing numerical computation and some of that include GPU clusters, and some of that includes just like especially networked computers. The second key difference is our work flow. A large fraction of our time is actually spent in the prototyping phase, as opposed to maintenance, and what this means is research, ideas come and go, and we don't want to invest a lot of time, engineering, something that might not make the cut.

B

Yeah so, as we just mentioned, we may need to prototyping and of course we want to build our infrastructure to best support that and to really be optimized for for this kind of workflow, which is very different from the workflow.

B

You would use when you're building say a web app with micro-services, and so a couple of key things that are important to our researchers is the English solid platform, and that is because, if they run their experiments and they behave like weirdly differently, when they run it for two times, we actually don't have any knowledge gained, because we don't know what actually happened if something happened in the experiment or something something happened inside of the platform.

B

At the same time since report IV so much, that means we have free hyukoh churn and the configurations that experiments run in change very frequently. This can happen like in a matter of like hours when, when someone is like, oh yeah, let's try it.

B

Let's try some different configuration, and so our infrastructure needs to be needs to be flexible enough to be reconfigured super easily, and so the way we're tackling this is by having a central infer team, which is Vicki and me, and a couple of other people as of recently that runs of a common platform that the researchers used to run their experiments, and so the goal there is to empower users to run the experiments by themselves without needing engineering support, but at the same time be shielded from the complexity of the underlying systems, which is like the GPUs and different cloud providers that we use.

B

At the same time, because we have such a small team, we also need low operational complexity. We want to set the thing up and in the best case, it just works, and this is where coop comes into play. So up until a few weeks ago, like organize, everything was just really key. In me, we ran a coop cluster across three different child providers, our own physical Hardware. We scaled up to 4,000 nodes where we burst up to 4,000 nodes and it went pretty well.

B

We had no major outages and the key thing that really makes this work for us is that coop allows us to ship infrastructure as internal products and what that means is. If infrastructure structure is a product, it has to have specific like product like qualities, the most important of which is, it needs to have a consistent, UI, that's understandable and doesn't and doesn't surprise users by default.

B

It should hide complexity, but it should also be composable so that, if you need to do something, that's inherently complex, you should be able to do it without ripping your hair out, and it turns out that the coop API is also it's actually pretty good candidate for it for us, and so we've actually started treating organizationally the coop API as the front-end that we, as the infrastructure team, provides to the end-users, which is our researchers.

A

And so building on top of the coop API is actually very nice. So coop API is our primary service layer and what this means is we can take advantage of a lot of the abstractions that are baked into the coop API to help us compose the different components in our experiments. So, typically, what happens is in the life of an experiment or a new research idea. We start out as like a tiny tip, tiny prototype that runs as like a single pod and the cluster.

A

You know you might have like one Python script, that just runs in your network and then you see how that goes, and so, as the idea turns out to be like more promising, we try to scale up to tackle more complex problems, and so what that usually entails is, as you tackle harder problems, you, the trading time scales up a lot, and so we start to have need for a higher parallelization need to think of different ways of distributing the training and also if the idea seems promising, we also will need to run many different.

A

Copies of the experiments may be running, variants of the idea or running hyper parameter sweeps so here's an example of a large batch of experiments that we've had to scale up very recently, so we, the exemple, is that we're training agents to play video games right, so we're running agents against all these like games on our universe platform and universe, is a platform of over a thousand video games that form a data set that allows our AI agents to learn to play different games and you can see there's a diverse types of games.

A

Some are like super simple and some are actually like fairly complex. So as we first start out right, we have just the vanilla set up. We put a video game, let's say a flash game in a container, and then we also put our agent in another container and we just like run them together in a pod, and this works super fine for like prototyping, because you can use the same setup on your laptop and then you can like ship. It off run it remotely in our cluster, but then it turns out.

A

Actually, that games are really complex, and so you know you can run this for maybe solving pong or something and then once you start running like into racing games or platformers, it actually takes our agents a lot of time to learn to solve it sufficiently, and so, unless you're willing to wait for like hours or days actually as like a very long turnaround time to play with a new model. So then we've had to change it so that we can train in parallel.

A

So what that means concretely, is that, like you, have one master process, which is your tensor flow parameter server, and then you split out your training in two parallel workers that interact with their own copy of the flash game, and then that way you can split it up and basically scale your training. Once we start your training. This way we can now tackle more difficult games and, as the complex games go, they have heavy graphics impairments and so on, and so they actually became the CPU bottleneck. And so what happened?

A

Was we had to split into the flash games had to run on their own in a separate host, and then the agents would interact with the flash game over the network and they still talked to the master parameter server to update their models, and so this actually means there's like a couple of intricacies in this setup right. So there's like how we'd network or like how we schedule the different jobs on this cluster is so that they can have high networking, bandwidth and oftentimes.

A

The agent might have like special compute requirements, and so it's actually pretty complex to set it up, but in practice all it took was just changing the yamo file.

B

Yeah so as you've seen for every experiment, usually we start out very simple: there's a single person running a single single job leads to a single pod, all as well, but then eventually we'll scale it up, and so the this, the the speed at which this scale increases over time for experiments are successful, can be pretty intense.

B

So like a typical typical thing to happen would be a researcher plays with an algorithm on a single core for let's say two months and then suddenly they realize it's a paper deadline and now they need to run all their experiments for it. For that paper in in two weeks, and now they suddenly scale up from one core to 10,000 cores which can happen, and the thing is that it's.

B

This is very hard for us to predict in advance and very bursty of course, and that actually means that that we can't keep our cluster size constant, because that would just mean like it would either be just too small, which means we can't run the big jobs or would be too big, which means it would be very wasteful because we have a lot of idle capacity.

B

So it actually means we have to. We have to resize our cluster dynamically.

B

So to do that, we wrote an auto scaler, there's also an official order scaler, but but we started doing that before the and actually goes to show that that just using the parameters of the kuba API, this works pretty well, and you could make it work. You don't actually need to need to have support from like core coop necessarily so what the autoscaler does. Is it gathers the state of what's happening in the cluster like what nodes are there?

B

What pods are what waiting to be scheduled like what pods can't be currently scheduled, because you don't have capacity and that's just using the the qu API, and then we actually provision new nodes exactly to fit to fit the the resource requirements that are missing from the current cluster, and this happens using using the AWS and as you're provisioning API s. And that means we have different from how you reduce auto scaling groups where they just scale up after there's a lot of CPU load for a while.

B

We actually have complete control over the types of machines that join the cluster and the types of machines that that do not join in the cluster. For example, we can implement something like by default, put everything on spot instances, because it's much cheaper to start that, except, of course, when people like especially opt out like don't put me on the spot instance. I didn't implement checkpointing.

B

But basically, these these types of policies is something that breakin we can barely very, very easily implement and very cleanly separated from from the actual logic that that does the scaling and I think it's fair to say that this is a result of of how the coop API is designed and that it basically allows you to to interact with with the primitives that are provided using your own custom logic.

A

So why are we using coop? So we've explained that our workloads are fundamentally different from maybe the common use case that could was designed for it, which is probably web services or micro services type thing. So it wasn't perfect out of the box when we first set it up a year and a half ago, and but actually at the most basic level. Kubernetes exposes very good primitives and interfaces that we can build on and Strax away, alot of the complexities of the underlying infrastructure, which is great for our users.

A

That may not have operational experience, so the flexibility goes both ways. It exposes a very clean API for our users, which are researchers and also for the maintainer switch our infrastructure team. So we can extend and customize the cluster as needed, because you never know when the experiment will lead us and so we're very excited to continue to invest in kubernetes and we're also super excited to see the new developments coming in 1:6 and also for the next year. So thank you.