Numenta Live Streams, 8 Jun 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Numenta Research Paper Review: Scalable training of artificial neural networks

Description

Review of the paper Scalable training of artificial neural networks and how it relates to our ongoing research on applying sparsity to neural networks (https://www.nature.com/articles/s41467-018-04316-3).

Additional papers reviewed to set the background for the discussion:

1) Rethinking the Value of Network Pruning (https://arxiv.org/abs/1810.05270): structured pruning using several different approaches, reinitialize remaining weights to random values.

2) The Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635): Finding Sparse, Trainable Neural Networks: unstructured pruning based on the magnitude of final weights, set remaining weights to initial values.

3) Deconstructing Lottery Tickets (https://arxiv.org/abs/1905.01067): Zeros, Signs, and the Supermask: unstructured pruning based on the magnitude of final weights or the magnitude increase, set weights to constants with same sign as previous initial values.

Structured pruning usually refers to changing the network architecture, like removing a filter or a layer.
Unstructured pruning is “sparsifying”, killing the connections by setting the weights to zero and freezing.

Broadcasted live on Twitch -- Watch live at https://www.twitch.tv/rhyolight_

A

Hello, everybody all right. We're live thanks for joining.

B

A

Twitch, they cannot hear me right now. We're gonna be watching Lucas over this paper. I.

B

Might just not if someone want me to promote the paper, it's gonna be kind of a pain, since this webcam is also pointing at the ball. So I just won't protect the paper. Unless someone wants you to.

B

Matt's handling that that's doing mr. Barba's, oh.

A

Yeah, the only the only thing I really need is his Marcus's camera at the on the whiteboard.

B

I mean for us in the room, if we want this up here,.

C

A

They're, just talking about they're projecting this paper.

C

Actually, it's.

C

That's how different working lottery tickets, yeah.

A

There's another paper: okay,.

B

I think we can start ugly here. Does so Matt's got everything covered.

D

D

C

A

Another paper reference focus, slowing.

C

Down I, just miss rivers before we get that, so is that clear: do that and that paper okay, the motivation. Why and.

A

It's something of.

C

So what we've been doing, the neural networks is regular brownie and you go routines usually to make a network smaller to fit in smaller, like Isis, so it's better to doing for a circuit by connects Mariana phone, and you can also improve actress by fine-tuning the smaller network. So they have this two mangos pruning, but pruning has been traditionally done.

C

Keeping glass collections or less layers depends if it's structure on the finish, I couldn't buy proven by removing a few connections, but I usually keep the same ways so, but we fought is that the final wait we got to using their natural cut, the weights that are optimal for learn the same weight and it just from a few facts? So that's traditional!

C

So this year two papers came out tonight, clear I, think subtype watch, then one is called we're thinking of a leaf network pruning, which is a really insert. Do you think he works with us, perfect pruning, so his throat that pruning is instead of cutting now a few connections and make it more sparse. He cuts now like an entire future easily or making a layer. So it's very similar to what we call a large its network, research, so I think this.

C

Is it guys, a fool, not true, and then he evaluates do I need is in time networks. So what can I cut down and still get the same result or even better results? So this is suspected pruning and the these papers that he shows that if he applies from the bruni he can get the same network. We said all the ways to random variables and train again and he's gonna get even higher a question, the initial network, so the difference here is that wait.

C

So what it shows is I, don't need a sign of it and we set the weights all. It means that final architecture. So it's very single for a neural network. Research between proteins.

D

C

Just their their prototype right, network of tech, research usually start from small, and then you start to add more stuff. In this case,.

A

D

Understanding why that works.

C

Yeah, okay, so these are the paper. It's called a lottery ticket hypothesis, which came run with that paper and one of you guys people likely and what.

A

C

Are doing they're working with obstructed protein, which is fun a paper from 2015 we I'm not sure, and what he does is like his sparse in the network. So after he finished training here this seaweed connections he doesn't need, and you just could've out, so that we're effectively make the network more sparse and the way he cuts announced by magnitude. So he gets the one with weights closest to zero, and then we move in and just keep them on to higher space. So, though, with weights close to 0 so different.

D

Than the first.

C

But the difference is that he also similar to this also shows that you don't need to find a way. So you can result in issue and you can have a needle mara, Kristen and forth by training the news, then a misfortunate yeah by training. The news pass that.

D

A

Better part was.

C

A little bit yeah, there's.

D

Like a D, that's all I mean one if it's on park, okay, another cheap watch, I mean YouTube. The initial benefits cutting back but was surprising, is that when you retraining.

C

C

Some points, but it's like a needle but.

D

Still it I mean you can even imagine a little bit better. It's sort of like kind of noisy.

C

Obvious it would get.

D

C

If you, even though they're pruning in it and resetting into the initial values during training that gradients at each neuron is not be different, it's not should actually get.

C

Yeah I think better part is like.

C

At some point, it's.

D

C

And now there is this new bit just came up so for all of these three things that you presented so.

A

C

Order together, you have to train a complete full yeah.

B

C

The training process is actually much worse, but it's much lower because you have to well. If you just proved it, you know you train the whole thing and you prune it and your days.

D

Are worse than it was before.

C

You train it. You prune.

D

B

C

Is actually much worse.

B

D

No no Punk I said cause. It least you have to do it twice. Yeah.

C

So, but what these are showing is that a lot of the learning is going on not on the way to here can reset the wait to read my post, and here it can reset the initial that, and it's still learning better so.

C

When the first network was trained, we presented the same.

D

And you surround embarrassed.

C

And here's just any new rhythm so when they met when the nest networks first initialize, it has like a setup.

C

They think it's showing that there is a lot of learning going on, not just on the weights, but on that actual connection zone, which connections were for my dinner and this new paper emphasize that which you think it's really interesting. It's called a super mask and what they showed is treating exactly so. Firstly, they investigate. Why do you cut off the weights by magnitude, makes a difference and what they show is that if you actually, they are actually cutting down the weights which were already going towards zero.

C

So if you, instead of putting the weights by magnitude, but if you cut the weights by by a Delta by how much they change doing frame. So if you keep the weights, which increase more and remove the weights, which were too crazy, more it's even better than cutting by night. So what's really important here. What these guys do exactly removing the weights which we're going to remove anyway, so I kept on training. Those weights were going to be pushed to zero they're trending.

B

C

B

C

Training downs. What you doing is: okay, the Delta will do.

D

You think you've come to zero yeah.

C

So what what it's actually doing is those weights were already going to zero? Just the text before and cut them off, so I don't need anymore, and so it also does it does. The same thing said: there's a unscripted pruning and while it shows which difference from this one is that you don't even need initial balance, you can actually raise that to a constant, and the only important thing is that you keep the same side. So as long as this sign is the same, you can reset this initial. It's after we apply this mask.

C

You can wish that these initial weights to any constant, which is the same sign of the way I mean.

D

They all have the same value or is yeah.

C

C

And what's more crisp about this picture is about if.

D

You there's a during training is a sign change.

C

The sign that it was doing.

C

So the master isn't here: they are not keeping their weights, which change size because they were removing this. They were moving their weights, which went alarm 0. So if it was negative, let's say and the final rod was like negative 1, it might have R 0. So this is the kind of place that's gonna be removed or if it was 10 and then went towards 1, it would be remove.

D

For the tournament, so it's positive anger, but then you're saying it's continued. Its moving.

C

D

Student yeah: it makes that zero. Your honor.

C

Did I leave that, but you.

B

Think, generally, whatever sign that had any initialization.

C

It's likely to have the same side, yeah yeah,.

C

And willing to assume that okay, so Ukraine is our network and then you decide which connections I don't need so I don't need any of this connection. You have a mess and you can apply this mask to a new network and there's new networks. You randomly initialize all the weights to a constant, both credential or you, initialize, all the way to your constant, with the same sign as the original and without any training at all. It can achieve 86 percent attrition against doping.

C

You just use it from in prison against up to 86% divers of memories and enemies has been class. So this was equal freedom agenda, so I fix rampage on Ellis amendment so.

D

What so you're saying what you're saying is: preserving the sign and the connections in.

D

C

C

C

21 cipher, then it's 92, so.

D

Now, you're it so phase, you can get 41%. You go from this collection.

C

So this Venus alarm initially.

C

So that the household ants so sometimes Louise can correct me if I'm wrong huh how so there so we have now.

A

We talking about our, how could we be so dance.

C

Fixed initial connection, so they randomly decide they're, dropping uniform random distribution, but they're fixed a nice little bit. The connections which begin set to zero do these, such as given the whole frame, and we also have boosting, which helps the lower its connection to be selected winners there, and then this helps maintain a more stable distribution of weights. So you corrected it.

C

What we're doing House of dance is that the connection are not changing at all, so we're not like exploring this space, and the other thing is that we are actually helping the low weight connections to remain alive because boosting is helping kids, so another option to be very aggressive on these.

D

Are the words where is this: where is the sparsity coming from them? You're keeping all the connections.

C

Visualization, though we have a sparse set up initially.

D

I thought oh yeah, so we started with.

D

So we just we are just starting with the varsity.

C

Yeah, that's partly. We start that specific connection there can draw Coltrane. You.

C

Know when they prune work percent, how big is.

C

These numbers are, and the Superman and the lottery tickets about 10% the super mask.

D

C

Think there are I, don't they.

A

Either are right, it's pre print and in our recent.

C

Paper, how pretty piece being it's army but as they forget.

D

So they're getting enough Furies lower, yet power work.

C

D

C

Think we can do better than both both of those techniques that we actually added drop connections on the fly and we should be able to get by maybe with less.

D

Than 10 percent it definitely does the 30 person recognized. So this would be. This is sort of the adding sort of our sort of reach out through the scheme to potential stand up skin elasticity.

C

A

C

Boosting and the thing is and to be doing in the connection we might need to allow the connections to network decay and I. Don't know how those things like helping the weak ones remain alive. That's somehow that maybe we just need them today. Yeah, but there's a part of the boosting is not helping the small waist yeah it's allowing the different units, so it may just mean increasing the nards. That's enough yeah! It doesn't have to be infrequent units they participate in the process.

C

It will also be changing the back propagation process, so you are you're helping them yeah, but this in the training process, and so they.

A

Get selected more and by the end, selected.

C

B

D

More taste, so it's.

C

Lana, it helps keep the stability of.

D

Boosting the unit.

D

From hanging out doing nothing now all set discuss and purpose in life, and it's doing something and therefore the connection is only valuable I think that naturally follows from that. But oh they have a problem. It seems like he's trying to know all I did was because to make sure you're using the resources in the brain or was like okay, we have these cells they're not really increasing to decreasing number, or we can even take advantage of all of them, but the connections keep coming about all they want.

D

Of course, if I after through some sort of epsilon from zero anywhere I didn't have this participation that we do good, like lock,.

C

Hands to zero. The request is grow intermittently in house events. Yeah, were you nervous, we're just a standard STD, but we don't know indicate so there is a way to actually count.

D

Toward zero, when you know you know.

C

So in the paper we have done, Louis separately have done experiments with, and it shows that we can actually in the glue a lot I'm down they put in.

B

C

I want to say, there's a difference between a weight of 0 and a connection. That's missing as you're crazy you, when you actually connect, enacted, freeze the weight, which is you and the other way about you just said to you, but the weight is not full right. Yeah.

B

C

We we freeze it to zero yeah yeah, because if you just set it to zero.

C

We are locking to do exist in the beginning when we start a lot and please let the weight of the case forcing them to do you're, not also using zero in the initial structure. Yeah.

C

But wait they came out when on you, it's not pot, and so this brings up to the paper.

C

I mean we expect this time to be the same, at least since this is important, but I don't know, wait in the same section.

D

In the second I.

C

Don't think so so in this paper over here, they're scalable.

C

B

I, finally, get to the actual paper.

C

A

Whole background.

C

So these networks is their own. It expires by the science, so they randomly select from that's like Ross, the stars in litigation, starting with Washington. Yes, already.

C

So they ran initialize, it's very similar 12 videos, but we do a uniform, random distribution and the layer and what they do is they like. This is a and.

A

It's brand-new samples, basically saying it is the same thing.

C

From the description of.

C

Yeah, but what is different in what they are doing is that at each at work you would select 30 percent, which I think it's like very aggressive after collections with smallest ways, Duke Ulis connections and they will randomly restart somewhere else on the graph. Thank you to connect sometimes start to connect. I mean.

D

C

They always have a fixed percentage, but they're constantly dropping in great big ones. So every book that Robert service at this modest rate and grown could.

D

You know recruit me earlier that in our model that that would be something you would do. Even you would grow new connections, but we don't do the random activity.

C

Sons, but not randomly yeah I, think nice yeah, like then drop into third percent. We've.

A

B

D

There's a correlation right now.

D

Well, let's make a connection for that recently.

D

So we have to, we have to sort of give it a shot, doesn't get.

D

Are you trying to keep certain Sparsit activity.

B

C

Doing it a little bit better doing it within the initial sparse.

D

Guy back to Jenkins.

C

Yeah yeah yeah: it's like how.

D

Big of a potential pool do we need.

D

We have a mess right now we could I.

D

C

C

This is brain like changers and they actually their paper. They talk a lot about their is it's very brain. Inspired.

C

But the work specifically on sparsity is the.

C

Mathematics and.

C

A

C

This approach, it can have a sparse network with about I, think it's five percent, but we have an immense collection and you can gather same.

D

Reasoning, any sort of the.

C

The big thing about the.

C

B

C

C

So then they'll throw like this close to you and even keep the ones.

C

So yeah, that's the paper in a tweet I mean there's not much to talk about the paper, so a lot of experimental or which I think was a bit disappointed that you.

D

C

Try this new work.

D

C

There's a lot of beautiful Nazis like experienced and I was just like man, you with.

A

Your life testing.

C

Boundaries and.

A

C

Trying to explain why, looking better, that's what booth it's a couple, tweaks it imagine you're one is what we already talked about. We could add, based on the activity.

B

C

Thing here is they're, not if they drop a connection from this unit. They could replace it by a connection to another unit. The number of connections in a.

B

C

B

Mother Melanie is not keeping it constant at all, so.

C

You could imagine try to keep that at.

A

Least, a number of connected early.

C

Life is a little bit like the boosting, so.

D

In our case, so in this case they have the same number of units and yes reducing the number of connection yeah. When we do the boosting or the stage of pool or equipment, we we have a spark activation in units, and so, if I look at the computational time of the computational power does do we take advantage of that.

D

Do we take advantage of as far as activation or do we stop it go through every unit, you've known that are all being active at that point on so the supporter for sparse matrices.

C

In the frameworks is not good enough to well.

D

I was coming on, it's like. Oh, they get down to 5%, pretty big reduction if we only got down to 10%, but we had. You know.

D

B

D

Don't work here, no sparsity blocks. That's why so you're saying that just the way they're set up today, that's something that can be rectified or is it like.

C

With our kind of sparsity, it's not a good fit for GPUs, so basically the deep learning community don't even consider that I've seen the papers where they say. Oh yeah,.

D

You could prune this way, but it won't get the GPU. So we're not even gonna try a little bit harder for us, because some of the advantage comes from the Hat. I mean right now, we'll be trying to like sneak into their bed and say we're going to do it. You're, never better yeah and now says the next grant. Do you know you can't do.

D

The benchmarks, the same: are they yeah so.

C

Right now we, you know, we are not.

C

D

C

Think I think that these papers are noon is like mm I think this is like.

C

C

Little bit more thought that, and when was this.

C

Yeah conceptually I, if do any hand, I.

D

D

D

Might be able to.

D

D

C

Very simple: you know competition that has to be optimized pretty.

D

Straightforward, so we're hoping the Clio does right, whose violence you know something similar. You know they're doing these networks but saying highly proven. Networks is highly through networks, actually better on these architectures and you're better. Overall, you know company like.

D

C

Less energy, less.

D

Power faster yeah, there's a huge pent-up energy demand money for the people who want to compete against NVIDIA.

C

I think in general, the GPU.

C

Architectures that are yeah.

C

I think the other question I had here was related convolutional, it's a very minor question when they were doing convolutional.

B

They didn't do this.

D

But they're actually trying.

C

Mmp and CNN's so on CN n 0, mmm peas from everything, rpms yeah.

B

C

C

I think they, the senior parts like small stuff if they focus a lot yeah.

B

C

Parties and they are working with physics, data so right and then just try and see and look at.

C

C

D

C

The beginning, my lunch, whenever electro tending demostración, have a sparse further than men's. Pathology and I also hold other important profits that are instrumented learning infusions. So he gives a lot of reference of brain wagons far.

D

A

All right, you guys hear me: whoops I should be show my email, iPad I, don't know which camera don't. Okay, a quick recap of that meeting. You want to see it. Robot guy I'll show you the robot at another time, a quick, quick recap of that meeting. What you saw there was a review of the paper that I linked in in chat and then a couple other papers that were linked from that, but basically other techniques for applying sparsity to deep learning networks. There's sort of two that this one. The way we did it in.

A

How could we be so dense, was by sort of initially pruning connections and basically establishing potential pools and then keeping them keeping those weights that same way, the whole time hold on the that's weird they're, not supposed to cut that I'm still struggling with my setup, but what this paper as far as I can tell is it sort of applies applies that weight sparsity at every time step because it could be applying it at every time stuff.

A

So that's definitely something we could do because we're if we're taking out the weakest, the weakest connections. We want to also establish the new connections and instead of doing them randomly like this paper, does: let's do it in a smart way, so I think that's sort of the direction that we want to go and that's why this paper was presented at the research meeting now.