Numenta Numenta Talks, 9 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: NAISys 2020: Sparsity in the Neocortex, and its Implications for Machine Learning Poster Walkthrough

Description

Our VP of Research Subutai Ahmad walks us through the poster he presented at the NAISys Conference in November 2020.

Most deep learning networks today rely on dense representations. This is in stark contrast to our brains, which are extremely sparse. Why is this? Are there benefits to sparsity? In this poster, we review how sparsity is deeply ingrained in the brain. We then show how insights from the brain can be applied to practical AI systems. We show that sparse representations are generally not subject to interference and are extremely robust, as long as the underlying dimensionality is sufficiently high. A key property is that the ratio of the operable volume around a sparse vector divided by the volume of the representational space decreases exponentially with dimensionality. We then analyze computationally efficient sparse networks containing both sparse weights and sparse activations. Through simulations on popular benchmark datasets we show that sparse networks are more robust than dense networks, and more than 50 times faster than dense networks on FPGA platforms.

Link to poster: https://numenta.com/neuroscience-research/research-publications/posters/naisys-2020-sparsity-and-its-implications-for-machine-learning

For additional resources on this topic from NAISys, you can read our whitepaper here: https://numenta.com/neuroscience-research/research-publications/papers/Sparsity-Enables-50x-Performance-Acceleration-Deep-Learning-Networks

You can also watch a re-recording of Jeff’s NAISys talk here: https://www.youtube.com/watch?v=mGSG7I9VKDU
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.

Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://tinyurl.com/NumentaNewsDigest

Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter

Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta

Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/

Our Website:
https://numenta.com/

A

Hey everybody welcome. uh My name is subati ahmed and I work with jeff hawkins at numenta, and I'm going to give you a very quick tour of our poster sparsity in the neocortex and its implications for machine learning.

A

I think we're all getting used to this new way of showing posters and uh trying to replicate a little bit of what it would be like if we were standing right in front of one another- and I was talking through it um so the place to start on the poster is the is to look at the summary which is in this uh panel label summary um and the main takeaways are kind of uh outlined in here, and uh they are basically that one is that most deep learning systems do.

A

They rely on dense representations, unlike the neocortex which, where the representations are extremely sparse and it's actually in the neocortex sparsity shows up in many different ways, and it's actually, um uh you know in the neocortex both that activations of neurons are highly sparse, as well as the connectivity is really sparse and that's kind of outlined in this panel, I'm showing on the right here, sparsity in the neocortex and the activation sparsities are somewhere around point five percent to two percent um and the connectivity between two layers of cells that projected one another somewhere around one to ten percent.

A

An interesting thing here is, um you know the many interesting things about it, but one of us that deep learning systems are nothing like this um and there the activations are a lot more dense, closer to fifty percent with rayu um and the weight matrices are typically about a hundred percent dense, and so the question that we often ask is, you know: can deep learning networks benefit from sparsity and if so, exactly how?

A

And here we're showing two benefits? uh uh Sparsity uh one is that um sparse representations are actually extremely robust, particularly when the dimensionality of the system is quite high and the panel on the right kind of shows the walks through this from a mathematical standpoint and some charts that I'm happy to walk through and answer questions about on the slack.

A

um It takes a little bit of time to walk through this, but you can show mathematically that if you as the dimensionality increases the kind of the the robustness of sparse representations and the stability of sparse representations increases exponentially uh with that dimensionality um in the lower left panel. Here, I'm uh show a very simple construction through which we can create sparse networks, and these networks are sparse and activations and connections just like in the in the neocortex, which is.

A

This is actually quite unusual in deep learning to do this we have a very simple construction and we can train the networks through back propagation in this lower panel. Here we show, through experimental results, that if you train such sparse networks, you indeed do get much higher robustness and much higher than dense networks, and I'm showing here the results with two different data sets. One is with a speech data set. The google speech commands data set and if you look here, there are results for dense convolutional networks, as well as sparse convolutional networks.

A

The dense and sparse networks both get about the same accuracy in the on the test set, which are these are competitive accuracies.

A

But if you look at the noise score here, we're showing the average accuracy over a wide range of noisy inputs, and you can see that the sparse convolutional networks are significantly have significantly higher accuracy uh to noise than the dense networks, and this is true, even though the sparse networks, this one in here has about 10 percent of the number of uh non-zero weights that the dense convolutional network has and uh the lower two rows show what happens if you try to create smaller, dense networks, um you know, is it the question there is?

A

Is it just the number of parameters or is there something else? And if you create smaller, dense networks, you can reduce the number of parameters, but the accuracy starts to drop quite a bit and so does the accuracy on noise. So this really has to do with the fact that these are sparse representations embedded in a higher dimensional system, and that's really what's getting you the robustness, exactly as what would be predicted on the math and down below. We show that you can get similar results with cypher 10, which is an image data set.

A

If you look at the first row here under zero percent, that's the overall accuracy on the standard test set and then we add different levels of noise, and you can see the accuracy is- is quite a bit better if you look at under 10 noise, for example, with the vgg network uh construction, where the sparse network is about 12 percent better.

A

The last big thing we show in this poster is that we've actually spent quite a bit of time implementing sparse networks on hardware architectures in particular, on an fpga, and the thing here is that sparsity can really give tremendous improvements in efficiency of deep learning networks.

A

As long as you have the right kind of hardware infrastructure and the hardware substrate- and you chose fpgas, because that allows us to really operate at the circuit level and really create circuits that are optimized for sparse computations and uh we've done this on three different xilinx fpga platforms, so the small medium, I'm sorry, small, medium and large.

A

The things here to focus on. If you look at one of the platforms here, there's dense versus sparse, uh the sparse system gets about 50x higher throughput uh than uh than the dense systems, and we get this in two different ways. One is that if you look at a single network, the the sparse network is about 10 times faster than the dense network.

A

However, because the sparse networks use so much less memory, we can actually fit five times as many sparse networks on the chip and overall, if you combine these two, we get a 50x speedup. Overall, we could show that. Similarly, you get huge improvements in energy efficiency, and we also show here some results on gpus, showing that the sparse network is significantly faster than dense networks on on on sort of large large gpu systems as well.

A

Okay, so kind of zooming back out. um You know a lot of the details are in these references here and happy to take your questions on slack and look forward to seeing you there. Thank you.