GitHub Universe 2017 - Featured Sessions, 25 Oct 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Pushing The Boundaries On Video Understanding - GitHub Universe 2017

Description

Presented by Manohar Paluri, Research Lead, Computer Vision at Facebook.

Pushing the frontiers of video understanding is necessary to get closer to general AI. This talk, which has a large emphasis on approaching problems with a multi-modal understanding, will detail Facebook’s video understanding efforts, highlighting aspects from platform, models, data and infrastructure tools, as well as exploring how partnering with GitHub has been instrumental in the company’s progress on this front.

About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.

For more information on GitHub Universe, check the website:
https://githubuniverse.com

A

♪ (music) ♪.

A

Cool., it's exciting to be here..

A

I want to thank the github organizers to invite me.

A

It's an amazing venue, and hopefully you have had fun over the last day and today.

A

Always honored to present some of the amazing work.

A

From my awesome colleagues at facebook., what you will see today, is basically a bunch of really awesome people.

A

Working hard on pushing the frontiers.

A

one of the main things I wanted to touch.

A

And I want to get everyone excited on is how we are pushing the frontiers of video understanding and why this is an important problem for moving towards general ai.

A

So hopefully, by the end of this talk you all, as developers are excited about this space.

A

And understand a little bit more about the challenges.

A

That we are facing in having general ai and understanding video.

A

So, let's start. first, and foremost,, taking a step back one of the motivations. Why we are working on artificial intelligence? Is we want to push the frontiers?

A

We want to make the field push cutting-edge technology in machine intelligence space,, but also make it accessible to everyone.

A

It's kind of fitting that I am here, talking in the github conference.

A

Because it's one of the themes there too., so we want this technology to be accessible to everyone.

A

Now, facebook and its family of apps.

A

Are being used by more than 2 billion people, everyday.

A

As part of this,, we have something called a social graph.

A

Now, you take that social graph-- and one of the things that we are trying to do is building the ai backbone to make this graph semantic.

A

What do I mean by that?? Let's spend a minute on understanding how we take the social graph.

A

And make it semantic, and how that connects to video.

A

when facebook started, it was basically from colleges and universities.

A

To connecting with people, right? so as part of that,, the social graph was people interacting with people.

A

And as part of that, using textural communication, writing, posts, writing status, updates.

A

and then moving to richer media, which is using photos.

A

To express themselves. now, of course,, we are in the video revolution, so the majority of the communication is actually still moving towards richer media.

A

And catching up with video and experiences in video.

A

That's the social graph.. You have this hard notes where people update their status...

A

Share posts, share images, and videos, and experiences.

A

What the ai backbone is trying to do is understand this content.

A

We want to make sure that facebook, as part of the integrity and safety,.

A

Preserves people's safety.: we want to make sure that your experience is enhanced and personalized.

A

As part of making the social graph semantic, the ai backbone is trying to understand that this particular photo,.

A

While it's a policy,, this particular photo is about food or you're, enjoying your vacation..

A

This particular video is about running, in that particular case..

A

This is the roof of our building 20., it's a garden, which is amazing., so that's the ai backbone, that is trying to make.

A

The social graph semantic. now,: why do we need to do that? and what is the advantage of doing it and thinking about it? This way?

A

the main advantage is:, if you think about accessibility.

A

So we launch this feature called automatic alt text, which describes photos for blind. People.

A

accessibility comes in the forefront, but developers inside facebook.

A

Can now think about all of this content and reason about it in a way.

A

That they could not before. the same particular nodes,. The virtual nodes that are created by ai.

A

Can be used to do better news, feed ranking to give results on search in a much better way and to make facebook a safer place..

A

So this is one of the fundamental reasons why we in applied machine learning.

A

Are trying to take this social graph and make it semantic in nature.

A

Now, coming to video,, as I said, a lot of communication.

A

Is actually around video and consuming video, producing video.

A

And so on and so forth. and there are some unique aspects of video on facebook.

A

So in this particular simple example,, this is a video recording of someone.

A

Experiencing the metro., you obviously have pixels overtime, that is video,, but on top of that.

A

You have this amazing amount of metadata that is being produced.

A

That is being given,, which is they are liking. This, loving this at this particular instance,, and there are comments going on around:.

A

"This is an awesome. Metro.", "these are the facilities.".

A

And so on and so forth., so not only do we have that video and the pixels overtime.

A

But we also have this very rich metadata that we can use to understand. Video.

A

And that's just one of the unique aspects.

A

and I honestly believe that understanding video is going to be key to connect billions of people to the content.

A

That is being produced., so if we don't build the right technology and the platforms.

A

And put it in front of the developers to [inaudible ],, we will not be able to understand this video content, and there will be black boxes.. So that's why I want to motivate all of you.

A

As to what challenges we have in front of us and what we are doing to overcome, them.

A

Let's talk about typical machine learning cycle.

A

Hopefully, some of you here are--, maybe many of you- have interacted.

A

With machine learning systems or developed machine learning systems., we always start with data.

A

we, currently--most of the state-of-the-art networks.

A

Or the state-of-the-art techniques-- are fully supervised or weekly supervised.

A

Or semi-supervised., what does that mean?? That means we want to add information on top of this data.

A

That lets the model, learn something that we want it to learn..

A

So it could be these images of a cat,. These videos of someone skiing.

A

So supervision is something that is useful, or this is a post that you might like or not like, and so on, and so forth.

A

So that's-- you add supervision on top of the data..

A

Then you select a model. you basically say, "I want to use a deep neural network, a [communication] network, or I want to use a simple, logistic, regression, or a sparse neural network or a [inaudible ],.

A

Any machine learning model that you think is actually sufficient or reliable for you.

A

To get the problem solved., you put all of this three together, and you have any platform that exist out there and you train your model.

A

in the training,. You get some training accuracy,, you basically test it,. Hopefully you have a test set.

A

that is mutual exclusive from the training set--.

A

No overlapping there..

A

I know many people have been burned by this..

A

So now you have a model,, you train it on the training data, you test, it.

A

You test it, the numbers, look good,, the you push it to production..

A

That's the deployment phase., that's where the real cycle- starts., that's the first iteration,, and to get to that you have done a lot of work,, but that's where the real big game, begins,.

A

The real awesomeness begins.! So now you have this full loop where you can test your model, interaction.

A

And see where it fails,, where it performs well, and you inform the next model, the next version of the model.

A

Using the data where it fails, improving the model., so this is the active learning loop and this is the machine learning cycle.

A

Now, the best part for us as developers and to make sure the rest of the developers can actually do this really, really quickly and reliably,, and making sure that all of these components are state-of-the-art, is what makes this job and particularly video understanding a challenge..

A

Let me walk you through all of these components with [ inaudible], video understanding..

A

So one of the things that we have built internally and we talk about this publicly- is once you take a photo or a video,. You actually--.

A

It goes to this platform called lumos.

A

What is lumos? lumos is basically a platform that runs on billions of images, everyday.

A

And videos everyday.

A

it looks at these photos and videos and tries to understand what is happening.

A

There are more than 400 visual models in the lumos platform.

A

So this is a computer vision platform that is used within facebook by a lot of developers within facebook.

A

and the models that are trained on lumos are built on above hundreds of millions of images..

A

You need these large-scale datasets to be able to train the state-of-the-art models, and make sure the accuracies that you have are very high for production, quality.

A

now,. Why is lumos so important??

A

If you think about the state-of-the-art networks in computer vision, their deep residual networks based on convolutions., you train them for a specific task., but the reality is, we as developers, when we build these models or platform.

A

We think of scenario 1 to scenario n.

A

But as users, or as people who are actually using this.

A

For various products and various experiences, they will come up with scenario n+1.. What happens? Then? obviously,, one of the things that you could do is train these models.

A

Typically, in 2012,, when we first started seeing the evolution of deep networks.

A

It used to take days and weeks,. Now we are reducing it to days and hours.

A

I'll talk a little bit about what our contributions are there in that space. coming back,, you have the scenario n+1., so one option is basically to train another model.

A

Now,, this is all good.. It will give you good numbers.

A

But the reality is,: we need to round these models on billions of images and videos., so you cannot scale this for all the tasks at facebook,.

A

For all the tasks in general., so what do you do?? The nice thing about these deep neural, networks, or the state-of-the-art networks,? Is they are hierarchical in nature., so you can actually, individually in the model, break it down to multiple steps..

A

What is the advantage of that?? So when you have a new task,, you can take the representations,.

A

The intermediate representations, at various levels for your task.

A

So you, as a developer now, can make a decision whether you want to go all the way to the extreme and train a dedicated model for your task or you use an existing representations.

A

Because they get the job done. to do, this, lumos actually gives you this capability.

A

When I say, "I have all of these tasks.. Let me take the existing representations wherever possible or build.

A

New representations when I need one.", and that gives you the sweet capability as a platform to find the spot.

A

Where you decide what compute [inaudible] problem and what accuracy is sufficient for you for your task to deploy and production and to put it in front of billions of people., so that is lumos.. This is what we're doing to make developers job easy in understanding images, in actually reasoning with videos.

A

And when I talk about images,, I obviously talk--.

A

One way to think about video is set of frames, so we can use the same existing technology on images for video as well.

A

So in lumos today we can collect training data,. We can train a new model.

A

And push it to production within few minutes.

A

This is really cool because,. Let's say,, you want to start with--.

A

I want a model as a developer to recognize fireworks.

A

That get uploaded to facebook.. You can do this within minutes where you can go and say, "I want to collect some data for fireworks.".

A

So if there are any posts, public of course, which contain photos or videos where there's a mention of fireworks as texts,, you can basically collect some data,, and this will allow you to collect data.

A

That is not fully supervised,. That's crowdsourced,!

A

That is basically using metadata around the post.

A

Once you have that, you collect-- obviously,, there might be a mention.

A

Of fireworks but the actual photo or video might not be of fireworks.

A

So what we do is we actually cluster them using the visual representations.

A

That we already have, and now when a human has to annotate.

A

They don't need to annotate each and every individual content,.

A

Which takes hours or days depending on the scale., so what they can do is they can actually use the clustering mechanisms.

A

That we have as part of lumos and annotate one cluster at a time.

A

Where, if the thing that cluster is looking good for their task,.

A

They can actually have all of these images as part of training set.

A

As positive examples., so we have now taken something like an abstract problem of having a model for fireworks to something very concrete where the data gets generated within minutes..

A

This is really crucial, because if you have the right data, you can actually train the model.. So in this case, we're actually going to go ahead and train a model.

A

And when you're training a model,, you always want to think about the final goal,, not that it's a computer vision problem or a text, understanding, problem, or speech or audio problem..

A

So what we have in lumos is not only the computer vision representations.

A

But representations for audio, speech and text which you can merge.

A

And then say, "is this post about fireworks?, is this video about fireworks?" and so on.. So this is a multi-model, and then,. You click the deploy button.

A

It gets deployed and it's part of the facebook visual cortex,.

A

Which is set off all these 400 models that gets run on new images and videos.

A

That get uploaded. and you can immediately see what is the live distribution., so you can see what this model is doing in production.

A

And see that, "oh, okay, the examples that are fighting.

A

With high confidence are actually fireworks." until this model is good,, and then you can also go and correct the data on the live preview and actually teach the model.

A

So,, you have taken a computer vision problem or a mission learning problem, and you have simplified it into all of these tasks and you have given the tools to the developers to make the best to use cases that we are not aware. Of., we don't know what is the significance of dropping at that accuracy by 1%.

A

And saving the computer by 50%. but as the developers, we'll actually know this much better.

A

So,, that's lumos, and we can take these models and apply it on videos. so,. This is an example where you take these representations and you take these models and run it on videos., and here you can see, live videos being scored for various objects,.

A

Scenes, actions and so on and so forth.

A

So, now we have at least scratched the surface of understanding, videos.

A

In terms of classification,, which is what is this video about-- in terms of scene, in terms of object, in terms of actions.

A

so,? That's lumos,, and this is basically a platform.

A

That is used by internal facebook developers and used for various experiences,, as I said, to make sure the facebook is safe.

A

The integrity, [inaudible ] problems, to make sure it's a personalized experience--.

A

You have personalized experiences and to make sure that, however, you're connecting with people, you're connecting in the right way and making sure your time spent is actually valuable.

A

So,, that was the data part., that was a platform which, where you can collect the data and train new models in an existing lane of scalable way.

A

We have also worked on modeling. so,. Here is an example of a model that is actually on github--.

A

The code and the model., it's called mask r-cnn.. This is a state-of-the-art network for understanding, images.

A

So this model actually allows you to do.

A

Instance, level segmentation on the cognition.

A

I'll tell you a little bit of what the capabilities are..

A

So here is an example. Output of this network.

A

That, when you're, given an image as an input, it actually tells you, where are the classes, which pixels belong to that class,, where it is,? How many instances.

A

Of the class are, and so on and so forth.

A

so here, you can understand how rich the model is.

A

In terms of understanding the real scene., the chairs,, the bottles,, the table,, the plate and so on, and so forth.

A

So,, this is one model that can actually do classification, detection, segmentation,.

A

And going further-- human pose estimation.

A

So,, one of the important things that we have noticed is.

A

When you're trying to understand images or video, the first thing is to understand people in the images and videos.

A

This actually exposes or allows you to build a lot of really exciting experiences..

A

Once you start understanding humans,, where they are, what activities they're performing,, you can do much better.. So here is an example of the same network,, mask r-cnn.

A

Trying to do pose estimation., so here you see the output example of the pose of the person.

A

And using this pose,, you can actually start reasoning about what activities people are doing in images..

A

This is an amazing amount of work from the colleagues.

A

At facebook, ai research that have put this model and the code on github.

A

Accessible to everyone, else.

A

so,, one thing I want to talk about is: we have made a tremendous amount.

A

Of progress in understanding images., what is stopping us from applying this in videos?.

A

Isn't it just like, "take these models and apply it on every frame of the video?.

A

And,, will it not work?", actually it does.

A

I'd say,, it's a reasonable way of thinking about video understanding as a problem.

A

You take these models and you apply on individual frames in a--.

A

And you see these amazing results., so here is another example of pose estimation on segmentation.

A

Of someone performing acrobatics., you can clearly see the segmentation works really well and the pose estimation works really well. so, yes,. You can get very far in terms of video understanding.

A

By applying image, models.

A

A

Another example here is a--. This is an example.

A

That we talked about [inaudible],, where we want to take these mask r-cnn like models and make it available for the developers so that you can run it.

A

On device in real time. here is an example where we took that network and quantized. It.

A

And made it super small-- run it at ten frames per second.

A

On your phone- and it can do real time, pose estimation of people.

A

So,, this is another direction which is very exciting. Research,.

A

And you can start making your camera understand a lot more of what it's seeing.

A

And building really exciting experiences there.

A

So,, these are all models where you take the best image models.

A

And you run them on videos or you quantized them and run them on your phone in real time..

A

Here is another example where we can take face detection and make your experience.

A

Of video watching much better.

A

so,, how many of you have bandwidth issues??

A

It's like--, we probably in the silicon valley, don't have that as a problem, but the rest of the world, many people in developing countries have this..

A

So here is an example where to your left is a person encoded with normal video encoding.

A

But to your right is a person encoded with smart video, encoding,.

A

Where you are actually encoding, the faces.

A

You detect the face and track and you encode that face in a different way, in higher resolution., but the overall bit rate for the video is same, but the watching experience is much better because you have lot more clarity on the pixels of the face..

A

So you can take existing image based models and go a long way with respect to video understanding and creating new experiences.

A

Let's come back to where we started.

A

so,, we talked about the platform lumos., we talked about how we can have the best data and best supervision..

A

We talked a little bit about the model.

A

But,, what about training? so, as part of-- recently, a few months ago?

A

We actually talked about it publicly..

A

This is the imagenet training in one hour.

A

for folks who don't understand--, who don't know imagenet, it's basically a benchmark in computer vision that revolutionized.

A

Some of these models in terms of image, understanding and deep learning.

A

Initially,, when we were training these models, it used to take weeks or days.

A

but,. The imagenet in one hour is an example where we did distributed training and the folks in applied machine learning.

A

And infrastructure teams in facebook have done an amazing job.

A

To basically bring that time down to one hour.

A

imagine, you as a developer, making a change and it takes you.

A

Days or weeks to know whether that change is successful or not., it's going to kill you, it's going to kill your productivity.

A

It's going to kill your creativity and it's going to limit how many things you can try on how many problems you can try.

A

Imagenet in one hour is a perfect example where infrastructure and developer time.

A

Can be exponentially increased by bringing down the training time.

A

So here we use two [inaudible ] gpu's and brought down training and model.

A

For imagenet to one hour., this allowed us to experiment significantly on the architectures per video.. So a couple of benchmarks in video are sports1m and ucf101.

A

These are, basically, tasks, where you're, given a video you're supposed to predict which sport among the 480 sports.

A

That are in the database are ucf101, which 101 action, labels.

A

So,, usually it will take months or weeks, but using the imagenet in one hour, work.

A

That is fully available on github, on open-source, that anyone can actually use-- we're able to bring that time down.

A

To less than half an hour or less than a day for sports1m.

A

This has actually helped a lot., the second part-- so, training,, the platform, data is great.

A

But what about data and video?

A

do we, as a community of developers, as people working on ai, for example,.

A

Have the right data search and benchmarks to measure ourselves.

A

On video understanding?, so we have found that the current data search have limitations.

A

They have limitations because, in these particular examples,.

A

You don't need to look at the video to be able to predict that someone.

A

Is applying makeup- or this is about someone kayaking, and so on, and so forth.

A

You look at the frame and you're able to predict 90% there., so you don't need the additional temporal motion or temporal information.

A

So how do we build data search that actually get us closer.

A

To understanding video? that is not going to just by looking at the frame.

A

We can predict what is happening,, what actions are happening., that's an important question and we currently, as a community.

A

Don't know the full answers. so couple of things I wanted to talk today were two efforts from facebook.

A

There are two data searches that we are going to make available for the open-source-- for the community at large to build on.

A

One of these is soa. I'll talk about it. A little bit.

A

And the other one is generic motions.. Both of these data searches are efforts where we want the whole community.

A

To help push the video understanding, frontiers.

A

Why these two data searches and what are they?

A

soa is one of them. super excited about that..

A

It's stands for "scenes, objects and actions" in video.

A

There are some really nice attributes of this data search.

A

I don't want to list--, they are listed here.. I don't want to go through the full list., but the high level here is many of the existing video data. Searches,.

A

They are small or they might not be fully-supervised.

A

or they are the outputs of previous algorithms..

A

But in the case of soa,, we made a concentrated effort and invested significant amount of resources to [inaudible] to scale.

A

It's fully-supervised,, there are professional, annotators, actually annotating these videos and it has various other attributes.

A

That, when released next year,, the community and the developers,.

A

And academic folks can actually build. On.

A

here is an example of what the annotator does.

A

So we have these short clips., it's 5 to 15 seconds clips, and the human reviewer,.

A

Human annotator, goes and selects a bunch of open [, inaudible] tags.

A

So we don't tell them, "you need to confine to this dictionary.".

A

They basically look at the video and they add information, tags for objects.

A

For the scene of the video,, for what actions are taking place.

A

In the video, and in a free form way., so their dictionary is open..

A

These are professional annotators,. The videos are short.

A

And it's a multi-target, multi-class problem.

A

And the dataset is large., we are aiming to touch 1 million and many, many classes.

A

For all the three dimensions.

A

here is just a histogram plot.

A

on the y axis is the frequency, and on the x axis is various labels.

A

Obviously,, unless you have a magnifying glass,, you can't see.

A

What the text is of the concepts., but what I want to emphasize is you have a very long tale.

A

Of the dictionary., so the most frequently occurring tag or concept in video, in object is a person.

A

There's no surprise there., but you have a long tale of the objects that occur in natural videos.

A

Of the kind of actions that happen., if you take a simple action like biting,, it's about a human or a person.

A

Eating a sandwich, but a dog biting another dog or a cat biting something else...

A

The toy, and so on and so forth., so you have actions and when you combine it with objects, you have a combinatory explosion of the kind of things that can happen in the real world..

A

That is one of the reasons I am super excited to unlock.

A

Video understanding because it gets us one step closer to the general ai, problem. and soa- is one effort where we are trying to decouple actions, objects and scenes so that we can actually evaluate models..

A

Are they really good at understanding video or are they just looking at some visual patterns and saying.

A

"This is kayak.. This is kayaking because I see a kayak and a person.", but the real deal is,. You need a person actually kayaking to make it kayaking..

A

There are some initial results and findings. we're going to push this as a challenge for anyone to compete.

A

To actually train models on top of this data.

A

This is the first time we're actually making the public videos available for others to build models. On.

A

We have some initial results on architecture search..

A

We have done a significant study on what models to build.

A

Should it be limited to rgb? or should we have motion information as well??

A

Should we pretrain on other tasks and train on this??

A

So there are some initial-- a lot of architecture, search and work that we have done. and the learnings-- the high level learnings--is a bit new.

A

Let me just say,: this is not fully scientific, yet.

A

Because we need to do a lot more experiments, but the initial learnings are the variants.

A

For doing action, recognition or understanding temporal information,, you need optical flow.

A

optical flow seems to help a lot. pretraining on large datasets and then training on this seems to help a lot.

A

And all variants outperform the frame-based model.

A

This is very exciting.. What that means is, if you take models that are build on frames or images.

A

And are played on video,, they fall short compared to actually.

A

Building models on video.- that's really good, because video understanding is not about looking at snapshots and making [inaudible]..

A

Video understanding is understanding. The motion over time.

A

And we have some really exciting results on models that take minutes.

A

But are super-accurate. but models that can actually run real time on your device, that might be less accurate. and those are the mix convolution models..

A

The second dataset I want to talk about is the generic motions, dataset.

A

So here, again,, if you just look at the existing dataset,.

A

They focus on activity recognition or they focus on certain aspects of video.

A

But the real deal is to understand the fine grain motion. Information.

A

So let's take a couple of examples., so this is falling.

A

Falling:, a person can fall, a panda, can fall, anything can fall.

A

So you want to understand: what does it mean to be falling.

A

You want to understand what does it mean to be jumping.

A

You can, obviously, in the natural world, in the real world, in a natural videos.

A

You will have people jumping or car jumping., but you want to understand jumping, irrespective of what object is performing that action.. So the generic motions dataset is decoupling. The information.

A

Or the action recognition, or the actions from the objects.

A

Here is sliding, for example.

A

and here is, obviously,--. Our presentation is not complete.

A

Without a cat in the videos., so here is a cat actually sliding and falling.

A

One of the exciting things for us when we are building this dataset is to basically ask the question, "given a short video,.

A

In this case, these are gifs,. If we have multiple actions happening in the video,.

A

Can we build models to understand this?, and can this then help understand videos in general?".

A

So this is the generic motions dataset., and there are some unique aspects of this dataset as well..

A

It's focused on motion.. It is generic to different objects..

A

If you think about jumping,, we don't want to associate it with just animals or humans,. We want to associate it in general.

A

There are other different aspects: of,, like I said,. These are gifs.

A

They're small,, so you can train models., you don't need crazy amount of resources,, which is one of the things that we are excited. About.

A

and dataset is multi-labels,. So when you have an example.

A

You not only have one label,, you have multiple labels.

A

so that you are actually understanding multiple things in one go.

A

There are some preliminary results and the exciting thing about these results are. You can clearly see a huge gap.

A

Between models that are looking purely at pixels.

A

And models that are actually incorporating, motions.

A

so flow is basically optical flow that incorporates motion,.

A

And when you combine it with appearance,, you have much better accuracy.

A

So clearly, the dataset is trying to push or trying to make you make use.

A

Of motions to be able to do well.- and that is one of the key things in my opinion- to understand. Video.

A

And when we start understanding video,, we start making our cameras,.

A

We start making our cloud., we start pushing the frontiers of ai,, because otherwise we will be starting--.

A

We will only get stuck in understanding one snapshot after another.

A

But never understanding the real world.

A

We at facebook really are excited to keep on pushing the frontiers.

A

But also make these things available to the developer. Community.

A

These are just a few examples coming from the facebook research group.

A

Or applied machine learning, group, or the infrastructure, group, and many, many other groups at facebook that are in front,.

A

That are in the center of the ai world, that are basically in the hands of developers so that they can also push this frontiers along with us., and it's not just us pushing them..

A

That's it. thank you., (applause), ♪, (music), ♪.