Numenta 2014 Spring Hackathon Talks, 9 May 2014

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Temporal Pooling Fireside Chat with Jeff Hawkins

Description

Recorded at 2014 Spring NuPIC Hackathon

A

Okay, we're good we're, live.

B

Out here, if you guys want to be part of this, I really suggest sitting up close, I'm not going to be talking and you.

C

B

Leave you don't want him.

B

So, as I said earlier, I was going to talk. I got sick and hot obliged this week, the last three days, so you don't have one sweet them. There I'm feeling honey tempered intestinal blockage which each once they clear it here we go so I got stay in hospital for a while to time.

B

So I was asked about temple, cool and so I thought I would just as real special movement. That's question can be helpful. Do you have a question for me, so I thought I start off with what the idea of cooling is a gentleman focus real basics. So imagine this has been done many times you moon is a waitress goals, open you're, ready to earn surgery and the recording fees element in the high level. Visual cortex and animate show pictures of this person and they find out.

B

There's a cell that responds to celebrity like I, go click, and so any time she'll picture Bill Clinton to this person to sell attacks. It doesn't matter what capitulated way, we're working on the picture the cell becomes active, and but if you look at somebody else, they tell doesn't matter so it's a girl friend zone and that those self exist. Sometimes they find that it's like okay, only when it opens looking sideways as a post poet. So now we can understand that cells being part of a sparse root of representation.

B

It's a very specific feature that comes with being over train, let's say no program, but you haven't bet you know, but their cells, like you, spawned you're certain types of faces with noses, and things like this so as part of us Marsha, stupid representation.

B

But the point of this thing that bill Klem cell is staying on, while your eyes might show the picture of a dope n to the person every time there is movement, probably the new innovations coming around one and if you show up it to third picture completely alliteration coming up on to put renal cortex. So pooling is the idea that you have all these different patterns that map onto a relatively stable pattern. You know so you could say here's that is the stable pattern.

B

It was one cell, but there's a lot of stuff and here's a bunch of different continents that.

B

And all these patents math to this day, one and so the stability hear them nothing perfectly stable and and these are changing without you, and so it's pooling the idea that you are essentially cooling you're forming these guys are all pull together to result. In this sorry.

D

We're going to get glare on the number.

E

One of you want me to turn no, no, no.

D

No I don't want to disrupt it. Go prepare.

B

That's the basic idea is fully it's saying. Multiple pattern get relatives, mobile, relatively changing to relatively stable and, of course, you'd be trivial. To do this, if you just say turn all these cells in the model the time, so we want stability with uniqueness right. We don't want this all some things, I'll stay on. That's true, but you know we want to be stability with uniqueness of a show bill. Clinton's liability meters are so the other one personally publisher to her Aniston and so.

B

Is that I? Basically, this is question about that's a term by the way they may not, but other blues tip so I.

B

Now we've opened down the 250 pipes and temple pulling and let's just talk about spatial pulling briefly patient pooling is the poor cousin, it's the simple one, but it doesn't do much of it essentially says it's facial pooling.

B

You could group two patterns together and say they the same thing but they're only doing because I share the same bits, so I haven't I have a picture with a bunch of bits on it, and I could just turn some of the kids off at some of the bits on that would be those too proud interviews very similar because they're sharing northeast and bits. So it's a very, very it's a Hamming distance type thing. Basically, there's a facial overlapping team. Two patterns can't do much with it.

B

If I took an image and I read and I shifted over a few pixels hyperspatial pulling, does nothing every pixels change that in Europe your host right? So this is like the simplest hyper pool you can do and even in the CLA we have a spatial pulling stuff.

B

It's an important step, but it turns out that even if you don't train the space cool and you use a fixed one, it works pretty well, so you don't get a lot, it's beneficial to train doesn't, but you know it's not the most critical thing, the critical thing for learning and infants in this. By the way, it's just an inference process.

B

You take a bunch of patterns and producing a comment out, but that's information, I'm also a current in temples, and the idea here is that if in yours, if I consider these sequences that these patterns over time then- and I can predict from one moment to the next- what's going to happen because for various reasons was that sequence has occurred for an occasional predicted. If I can predict over the series of time, then we basically the brainless name. Evolutionists said these predict abilities means they must have an underlying cause.

B

It's the same, therefore, I will pull the ones that are predictable. So what an unpredictable I want to have something stable up here and if it's not particularly that I shouldn't, do that, because it's something different, that's the basic idea of temporal it's a numerous scientists have come to the same conclusion that this wasn't the way.

B

If you look back in the world today, I it's not where they started from oh and thought about this big inning, but the party people that have come to the same conclusion that temple pooling is really how it was all inference works entering this is the key. So when your eyes are moving around them or something it's not shifting an image, that's not what it's doing! It's a pletely new pattern. Every time your eyes move completely new, it's distorted! It's! It's not a transformation! It's a different pattern.

B

So, as you Sekot over my face or my body you're, getting a completely different set of things, but somehow your brain is able to say predict from 12, not evil. What's its kind of what is going to see and if it does enough of a prediction, it says: okay, I wanted a stable counted over here. For that, that's Jeff, or that's this way to work or something like that any restaurant owner is.

F

Well, that's a good question.

B

So you should hopefully have at least have a guest of that tail, because I said look as long as I can predict the next element here right then I can prediction that supplement that I want. You took the pooling, so one way to predict the next elements. Learn sequences of these things right and that's that was a there's. There's two basic types of temple: court.

B

One is the one we've included in the CLA today, which is based on the sequence, memory of of the ceiling, so I'll just call that we'll just call the temple point based on high order sequences.

B

That's the first type and there's simplest way to think about this way. This is not what's going on when your eyes moving okay. So let's don't think about that one, but this would be something like imagine. The simplistic example here would be on melody and these are select notes in the melody and it's silly, the sooner it. But if you've learned this melody connects off the next up the next on the next on the next hour, then you might say: oh, that's, that's a pattern, I hurry, or is that here's another one?

B

Here's a I'm going to say I'm going to say a word variation kept. Now you probably don't recognize that whatever shindig that's three syllables, but if I reverse the order who might get dictionary right so dictionary is a sequence you've heard before you were able to predict and you'll form a stable representation for that, but area should dick does one? Do you like the hell?

B

Is that- and you ever learned that before, and so it's not going to former state representation for their it's very it's very simple simple as that, so a high orbit sequence is a way that you can learn the sequence of these things. As long.

G

B

Able predict, then, you know, you've got the same thing up here and that's what we want to do. The second method is the one I've talked about in the recent meet up I'll just called sensory sensorimotor sequences.

B

This is like with us a comment, so if you look at the image or set anything and your eyes are moving around you're generating a sequence of patterns, but it's not a repeatable signals about it. You don't always look at the same order, keeping our ability different.

C

B

And you say: well, that's not a highway. Sequin top in the brain learn a sequence there and the answer we understand pretty well how this works. Now. Is that if you, if you have the changes in the center pattern, use it eraser, you can have to change in the center pattern.

B

Plus you have a copy of whatever motor command was just executed, copy of motor commands, meaning I see you haven't the cortex gets it then you can make the prediction and I'm not going to walk through this right now, but basically we've argued that in the cortex this is what they are three is doing, and this is what layer 4 is doing. These are the two basic types of sequence member you can do.

B

You can have a high water steam looks like a melody or you can have a city which is not a high or to see because, as you can, it's not always repeatable sequence of elements, but if you know what the copy of it, you know, what behavior you do like I knew my eyes and relaxes amount to write. This around I can still make the correct prediction, and- and if I can make the correct prediction, I can form a stable pattern. Up here did I answer is.

F

I'm still is two things: one is I'm trying to parse out where stability comes in and then the other part is where does invariant a while.

B

I haven't told you how we generate this I just said: it's gonna be okay and the other one was in bearings. Well, the advantage I one could argue that in variance is justice process right.

B

Imagine if I have a stable pattern here that stabled matter, where I'm going on the image and how my eyes are moving. That's invariance right! It's a doesn't, really matter what the time is. There's there's there's a lot to this. Well on your generalization. Got the business isn't about the hierarchical kind of stuff. I! Don't want to go there with you. The today's just talk with us to talk a bit about what tempo polling is and how we are going to implement. So that was the first part.

B

Is so I just thought, it'd be good to everyone understand what the hell pulling it I. Don't you know that that's what this is a poignant. It's multiple patterns get mapped to a relatively stable pattern. I think relatively it does not be perfectly stable. The parts of it keep changing. It's all about increasing stability as you go up. The hierarchies increasing stability over the tune.

B

It's a very, very important concept. It's basically.

B

This is the mechanism by which you learn to recognize that everything, whether it's time we touch in human someone, and now we can say, we've got the spatial pulling, which is kind of primitive, doesn't do a hell of a lot temple pulling is the key to making all that work and then there's two types of chum people, and that I believe at the moment that out those of it I, don't think it's going to be any other I, don't think, there's any other type of particular memory in terms of how you're going to do is I.

B

These are two fundamental method: hi or sequences and sensorimotor successful, as you have to bring all the way doesn't work with and everywhere in the rain. Your way forward and its really pretty much and that's what's going to happen. Yeah.

H

That, second one, it has to not be a high order. Sequence right.

B

It doesn't have to not be on our second, it could be a high of our students, but I, don't think, would fail. Those are cities, it turned out to my eyes, always economy, exact, same direction.

B

Then the mechanism we think it's going to work for layer four, would work but also mechanism of their.

H

Food so but its lair for doing any I ordered sequence memory, or is it I.

B

Would say it's either high or signal, and maybe someone who knows just to talk me out on this one. It's it's not sustained, given this I can take this. It's saying, given this plus the motor command, which is about the magic you I can predict this. Given this cost about humanity, actually I predictive, you know in the CLA, if you notice, when you make a successful prediction, it's much more support by. We are with oneself economy, so you go from the column of representation to a seems, only converse, you get a very sparse representation.

B

When you make a successful prediction, you get very sparse and that's we see that in the brain, so it doesn't matter which of these you're doing when you're in this one or this one, you have same basic affect the next representation. If you're able to successfully predicted to can be very sparse in one case, you can predict it just by knowing the sequence, the other one you had to say no I need to know where I was plus the motor command here, where I was plus the motor command.

H

History there so, but that look at where I am plus lurk man for that. Third, one is going to be affected by where you were to snow in ways that we not remember to know that we wouldn't be, but it would be in the first.

B

Yeah this one, basically, so they could you write in that sense. Okay, I'm, misunderstood your question. Change! Sorry! No! Yes in this one, you could be affected by the entire sequence right. What stayed here could be dependent or something that here, but in the second one here, actually not it's. It's really basically saying no! It's like you know what I now, where I am now and the statement now and where I'm going to go kind of it occur.

C

Well, so sorry, Jeff, you know just good layer for effectively represents a state machine of legal transmit legal transitions between states. In the current series, st.

B

Layer for represents legal transition to stay true to the current sequence, so I'd be magistrate machine, a.

G

B

G

You know anything that sounds.

B

Like it's, it's a.

A

Disgrace discreet.

B

Thank was the world's looking for thank you for life ins. We go is over. Thank you. Thank you. Super touch kind of the world's like that aim. It discreet it doesn't make me comfortable. These are not discrete. States trains are distributed, sparse distributed states, and so he doesn't be careful about falling into that trap of thinking about okay, here's a state machine. I could put them on a graph. It's not like that. It's everything all the time is there's a distributed pattern that some parts of our be predicted in some parts can't be predicted.

B

There are always novel representations of never exactly the same, but they have enough 50 common that the CLA continued support them. You have to try to always expunge discreteness or we think about these things. It's always fuzzy, but nothing common. That is good if.

A

You're Mike stop working.

B

It didn't fall for my head at all. Oh.

D

B

D

It must be batteries.

B

Mommy blog, we can see it is actually going to.

D

Get stops again pick our imma.

B

First, gonna get the microphone to work and then.

B

So you know sometimes you can select conceptually for well. You could say: okay, final x games, you may help you think about right, but you gotta be careful not to not to start thinking about that way too much about some of this. We don't really understand deeply. Oh I, don't know have all of us, but you can definitely say that it's very rare that these would be very exactly repeatable things.

I

This might be a silly question, but you have any evidence that different neurons in a single layer of a single region are synchronized in their firing.

B

We have evidence is any that red neurons in a single layer in a single region are synchronized to the front. So you talk about synchronization as in the firing at the same time or they're actually spiking. At the same thing,.

I

They're spiking, at the same time, okay,.

B

Yes, there's lots of evidence of synchronized spiking. You don't use any of that in HTML, CLA, three. Okay, at the moment, we think that is a plumbing issue for biology and it's not an information powering. That is, that is not an opinion that's held by everybody. Some people think the synchronization is the key to consciousness and all the smell of the stuff. I think it's bunch of nonsense, but it could be. We don't know, so we don't need to synchronize the actual spikes reading the house spike sincerely right.

B

We just have to say all these guys are firing at the same time, but the good reason to suggest that actual spiking them there on something we don't do is necessary to get real neurons to actually recognize coincidences. At the same time, yeah.

E

um You're, coming about layer for being a copy of the motor command coming in there yeah. Are you saying, in the general case, throughout cortex, you believe that layer 4, has projections from motor for length smarts.

B

That's what record everywhere? Is this really true that leg for is getting accomplished, motor commands out and round? Yes, there is evidence, and but I need to be more a little bit more. You want to know that I need to get a little bit detail. Okay to imagine, have a hierarchy of cortical regions and I'll, just so too. Okay and here you're some sensory input down here it might be preprocessed by some cortical, some political or that, and it then comes in to this region, their role everywhere as far as they bought their.

B

There are other things down here that we're doing generating behaviour, and so in brain these fine, these are subcortical Star Trek right in the brain. These could be anywhere from the spinal cord to the brainstem tube is becoming Leo things like that, it's pretty cool interest and so on, and these guys were paid. These are your reflex reactions, you're, walking everything you do and these guys generate outputs which make things happen and everywhere you go. You see that these guys split off the same action split up in the calendar quarter. They also post a comment.

B

It doesn't not so the general rule is the first region in the cortex. Your.

D

Mic tied again: okay.

B

The general rule is this: mom yep.

D

B

General is that if you look at the first region the context every one of these things either gender and behavior will split their at rons and you'll. Send them up here. So this guy gets both sensory and motor. That's the general rule and I think there was some discussion of the nice book by sherman, jewelry or naturally, but very sherman position show mango this news, demonic what they have. You can talk about this very nicely in there. Rafa diamonds do this and then then next question is ever be to the cortex.

B

Well, I think the data sure is a lot of fun. What we do know is the following: is that some of the lower readers so get some of this stuff, but maybe, as you go higher up a dozen, but we do know is it follow. Me is every region of the cortex. When I talked about this in my meat of talk recently, everybody reason the cortex has cells in layer 5 that generate that basically project should tornado slipper.

B

These cells project and projection ease generating things have these are motor generating visual motor traders and so every every region. The cortex has an output which impacts these lower level. They don't drive cells and look like neurons from cell I mean dumped our muscle structure. They associated the link to these guys and so every reason the cortex idiot motor complement, which goes someplace below, but also these things then project off to the next region. So you can look at this the same the dis region, even if it didn't get this pathway.

B

This region is getting a motor copy. Here, it's getting a copy of the motor commands, remove region below it as well as sensor data from here, and that's exactly what you say: we see a direct path and a indirect path like that, so I, even a mite, go higher open. The cortex I could say: yes, it's getting motor behavior, but it's getting them from the region's forward and below, so that that sort of hierarchical parallelism exists but subcortical hands in the cortex.

B

So I think that's a that's the evidence and it's a nice nice thing and there's no evidence contrary to this. So, yes, every region is getting some copy of motor commands across sensory and but even that's coming from somebody that was in.

J

The cortex yeah so in Egypt regions, spatial GUI and temporal cool, it happens and then it goes up and then habita.

B

Cares what I think is going on and I talked about this basically I think I, don't remember where I'll be going around, but if I look at a region with this breaking of a couple of spots, you can say here's layer, 41 students by three here's way for and then let's say, there's a five and six down here. Anybody comes in basic is the basic schematic are. You can becomes in here mrs. sensor data, and it's motive is motor cotton or bikinis all right.

B

So that's what goes into labor for that's the primary pathway and then personally or projectile 83 and my screen projection a parsec. This is there's a canonical circuit and the cortex is not simply this way how textbooks describing. What I think is going on is that you come in here. There's a spatial pulling going on when it comes in and inverting sequences.

B

Is it going to sensory motor sequences.

J

What do you say very secretive, that's technically, not temple.

B

Pulling at it's just learning the transition temple pulling is forming the stable pattern at the top. This is just learning the sequence. Not just her question was you know? How do I come to stay will happen. This is just like the transitions like okay, I. Don't want time to count of three predicting right, then what happens and you get to the when this goes a live. Three, you do temple pulley, / spatial pulling and then now high-order sequences.

B

So what does this mean? This means that if I can predict the changes in the input due to some motor command, then this this this layer will do a good job of predicting it and then I'm stable presentation of here and now, but if I can achieve the change we could pass through. So you've got two choice and get two chances here. This is happening pretty much.

B

Every region you say, can I make up kind of build a model of the data from the from motor commands and if I can great, then I'll look for the changes that can't be predicted. I thought a command or, if I can't believe. Indeed, even a modicum and I'll try to find all the changes here. Basically, this guy gets the best job and the remainder to pass something and the media content to John. So.

G

That's the basic idea, miss you repeated.

B

Over and over again she caught the hybrid. Don't you about.

H

The connection you talked about how the evidence for a lot of regions they're all the reasons that we can see how get sensory and motor input right when thinking about it, the nose right, your eyes are used to cut a lot and what you see the heads a lot on your motor behavior, the eyes. Yes,.

G

H

For your nose, it matters less right. You.

G

H

Smelling: what's an environment you know effect that that much do the olfactory connections tend to have weaker motor battery.

B

Cortexes of action is a sense, it's very poorly representative in cortex, and it's not already is what you're going to argue about this, but mostly on the way I do just pull faction is absolutely nothing to do this I. Don't you don't have it it's like an old census, very poorly, representing projects. There is no primary or factory cortex. In your context, there is a whole factory cortex, but its not New York. Our next is it's a different structure, almost completely so the very little temple to data processing manufacture.

B

But you see, if you sniff again, you spend again it's not the sequence of patterns. It's just over and over again, it's almost pure spatial patterns. If you're trying to take one of thousands of chemicals combine together some spatial pattern- you probably spatial pullin, probably pretty good way of recognizing motors but I, don't think its temporal all and son in some representative quarter hierarchy. That's you know, think about its rarity. It's very driven the sense for us. We don't have much high acuity.

B

It's like a it's like very visual and our emotional sense, but you don't sit there and go. Oh, let me walk through the smells and my kids, my garden and I'm visualizing. Not y'all. Do you smell something else? Oh yeah. That was my god. They.

H

Made it a dog's ever work, no.

B

I, didn't dog is the same thing. That doesn't mean it's not a very important sense. It's just it's not a temper sensitive. It's not.

H

A cortical, sighs, well I mean dogs are trying to locate something they their emotions do affect their yeah.

B

But it could be as simple as a you know, Paramecium turning following the gradient, it's hey! It's! If you think of all the senses, smell is the least it.

H

Was not even your question? There's no question is more about other some senses where motor matters more and you tend to see. Voter connections well, should be more strong.

B

Well then, it's only one could argue that in auditory space one could think like, oh, maybe, all four spaces less motor behavior right but I don't know if that's true I mean like what mostly what I'm hearing right now is based on my own bit, my favorite. So I'm not surprised to hear you know if you ever notice that, like a different listens like you scratch your head, you know how loud that is it's really loud, but you don't hear it at all.

B

You don't even because, but you don't sound occurred without you wiggle your finger on your head. You drive you crazy. Oh my god has been good. The.

G

B

That you did it yourself, Joey the nose. So that's evidence on your face. It's really helped additionally, Jeff.

F

Chicken you can put the mic down. That's how I give.

D

Out it I give up this without.

G

Yeah, this is not.

F

Gonna work at all just talk loudly. The only I got a mic right here. This is first.

B

On my phone, let us back well thatís hackathon: don't we got to redo the whole talk? I? Don't.

F

B

To do this, one okay.

F

B

G

D

At all, Jeff isn't the associativity of the smell handled by the neon for thank you.

B

Maybe I, don't you know I, don't know too much about it, but if there's no, you know, there's a did. A whole well mapped out pathway for somatosensory and auditory and vision. There's nothing like that open of action! I! Don't you! There are connections, obviously to the cortex, but there's.

E

Nothing at all, like.

B

What we talked about sure.

E

But yeah I mean there has to be enough for you to do the association of smells look.

A

Please yeah all right.

B

Yeah yeah I, just I, just think we should. We shouldn't focus on the faction what we want to understand how the cortex works. That's.

F

B

A fruitful exercise, that's my opinion and a faction people hate me for saying that.

D

Happen we can't even tell who dealt it.

C

F

C

Can that be generalized as a attention changes can.

B

The motor community generalizes attention to I, don't know I, don't think I. Don't we had a long conversation about some of this earlier I think would be unwise to go down that pathway right now. I, don't I, don't think would be fruitful to have that conversation right now. Okay, I I think its simplest way to think about. It is like there's a this. The part of their brain called the superior colliculus that moves your arms.

B

Ok, that generate a generates outputs with goes to the muscles that move your ocular motor muscles and those output literally get copied and sent up to the cortex. It's literally that simple just and it's called an efference copy.

B

So let's not put anything else on moment. It's just physically a copy of a motor thing and by the way by the basal ganglia and it might project to the brain stem and which might projection the spinal cord, and so a copy of this gets up. Well, this is the more higher level motor command right. It's done here.

B

It's driving a muscle, it's just something subcortical that is basically going to make something else happen somewhere in the world and and so we don't, the cortex can't know about all that stuff can't know all that crap, the top it has to have some way of working, regardless of what the rest of the subcortical stuff is doing. Yeah.

J

This question turns back to something you said earlier: what's the difference between a neuron firing, a neuron spiky all.

B

Right in our models, we we have several states for the one is it active, which would be typically associated with a real.

G

B

Of real biological that would be generating spikes, you know what spike zone. Oh, you don't know. Spencer, oh boy. We.

A

G

To spend a lot.

B

Of comments, but a real neuron is a cell and, as a part of the cell coming out called the axon, which is such a risk in tune and and if I look at if I, look along that acts on a spike is a is a temporary spike change in the voltage, the differential in that, and it moves quickly down here and so in a Cell each other. It's all its output, it as a spike lotus pond again.

B

Another spike typically do a train of these guys going down the axon, and sometimes it's more sputter spikes and there's rape to them, and things like that. We don't want that at all. We basically say: okay, any comments generating spikes per arm. Okay, spunk areas, its binary on. We also have another state which almost no I don't think any other neural network model has which is wheel on when the cell is not generating spikes, but it's depolarized. We need a the cell is in a state people.

B

It's in it's a voltage level. Is it's raised a little bit, so it's almost ready to spike, but it's not spiking, that's our predictive state. So this is something that we we do so we have a couple states, it's nothing. It's the polarized, it's active and all I'm saying is that we do not distinguish one. We do not model spikes, we don't models by timing, respite latency or spike to those patterns and things like that. Some other neural network models. Do then that's what I meant by that? Ok, no more spiking, okay.

B

So what we haven't talked about yet and that the purpose is target first, we'll describe what temple pulling it. We want to somehow get stable patterns away, so player for can make a good prediction. I want to get stable patterns in layer, 3 inflated, three convictive, a good prediction. I want to get stable patterns in the next layer that in projective and I want this stable. If I can make a prediction, I want to be able to stable the neck region or more stable and actually over and over and over again, whatever the pap.

B

It doesn't really matter which way we're going. But as long as I'm going upward, somehow forward I want to say, if I can predict sequences, then I want to have more stability. The next 11, that's what tempo cooling it separate from spacers for the sequence memory. It's the next thing. After that, the thing we added this last year was realizing that temple pooling is occurring between layers ourselves and there's evidence for this, and it's never occurred to me before think of it. That way, but so, for example,.

B

You know there's these things called simple cells and complex cells in in visual cortex and other cortex. Maybe you've heard those terms simple or complex cells, but as for example, you'll see cells as you go as you go in this direction.

B

You'll see cells that start out that they, they represent, say individual cortex, a cell orientation like this and then you'll see the cell response, no matter where it is in different patterns, so the cell is fun if a little mining, your visual field, it says: okay, well, that's how we respond to the lines they are. They are they over there. There doesn't really matter where it is: that's a complex cell, so you start out with this online orientation.

B

But now you have one that where they go from layer to layer, 3 I find yourself, it represents it'll, be active through any of these positions. Well, that's temporal pooling, that's basically saying I'm. Taking all these different patterns and the cells going to be active, regardless of where it is. So. That's that's. Essentially, the output temple pooling that would be going from I would argue from going from Leah for delivery. Then the next thing you see is you see, cells that are that are active only if thats, that sequence of patterns goes into particular direction.

B

Those are directionally sensitive, complex cells that will be output of layer 3. That would be essentially saying. Oh here's a sequence is, is a high order sequence. This is I or sequence. It's moving in one direction: it's not moving the other directions, not jumping around, so I might find in late for cells and say: okay, any where that line is I, don't care I'm going to respond active, but three I might say: oh here's, a bunch of cells only when it was in that direction. That's consistent with this basic concept: okay, yeah.

F

So, on the question of invariance, so specific example being melody playing different keys right, you should get some in various now. Is that the same high order see.

B

That's particular problem about melodies and different keys I'm going to punt on it's a hard problem. I haven't really quite figured it out or I had various hypotheses over the years, so I I don't want to be wanted to uncomfortable state trying to speculate on it right now. I could have some so there's lots of things. We don't understand, okay, so, but it does explain a lot of other embarrasses. Okay. This is a form of events right here.

B

It doesn't matter where the client that's a form of embarrassed, so the tournament barents and very reputations I use it in the blog on intelligence. You know it's a concept, but we have to be really precise. What we mean by it for going to use you so I prefer to talk about simple things like stability and sequences and prediction that are really hardcore things. I could say exactly what's going on there. All right I want to wrap this up. So I want to talk about how it is. We actually do temporal pulling.

B

Okay, exactly we don't know, but we have some speculations in the original in the original ceiling white paper. You talked about a mechanism for doing it and we implemented that mechanism. We've now abandoned that mechanism and I'm just going to give you our current thinking on the lumen. Okay and I've done a lot of reading about this trying to find biological evidence and tell us exactly me how much fun I read on told it to get about. A month ago, I.

F

B

Years I think I, don't think I registers I wrote and it was very interesting. I read through it and I said: okay. Well, what parts am I getting free, massage? Oh, the one maniac sure was pretty good I thought, but there was the part where I talk about temple pooling I didn't use the word type before I didn't know that name we didn't know about.

B

Some are stupid representations, but even like columns in Miranda stuff, but we talked about I talked about sequencer me and I talked about the concept of temple, calling about forming a stable reputation for sequence and I, said into Indian on intelligence. I said you know what we know. The brain has to be doing this, but there's many ways it could be doing it.

B

We don't really know enough to know which way it's doing it, and I said I will just described one just to show you that it's possible- and I described one in the book- is in chapter 6, where everyone gets lost. This is the part you really good one and and it's good, because the thing I described it almost certain is wrong, but but.

H

B

Say, I, don't know it's done, and this is just an exemplary that could be done. So even back then I didn't know, a temple pulling I had no really idea how it would implement and then our first implementation. It did it one way and now we have an idea for how to do it, and let me see if I can remember it right here. Sometimes it takes a lot, but these things click back into my hair. Let's.

G

Just say the following:.

B

I have what I have is a series of columns you can think of this as later for and some region of the cortex and then I have another series of columns. Another CLA, if you will, we think of it as a three or maybe the two regions of hierarchy. It could be a better scene, but basically the information is coming in this way, and it goes here and I want to I want to do temporal pulling here from sequences here.

B

Ok, I'm, going to learn, I'm, going to learn a predicted sequence here and I want from a stable representation here that clear. That's what that.

G

We're pulling is so.

B

Let's just talk about the situation where I can't temple pool what, if I can't learn in a sequence here now remember what does it mean? I mean I, have a sparse representation here in column, space and some sales active and that's changing all over the time. I can't draw that picture out of you afraid today, right I'm going to stay, but this guy is not predicting. Then we say all the cells in the active columns are not write. Something like get me and and what we're going to do here is we're going to.

G

Say: okay, this guy didn't.

B

Predict is it could all these columns of bursting and what we're going to do is this kind of pass goes through. Literally, these are the dishes later three. This is platform. These are literally just the same. Columns are just communing up in that region, so we're going to pass those through. So this is the simplest situation that you consider is saying: okay. Well, if this guy couldn't predict just pass out to the next guy, he's basically do nothing.

B

This isn't Noah right, but now that's easy, but now what if all of a sudden I could predict the next summer? So let's say: let's just a little bit and we're back to here now I am now, let's say: I have a new input and I'm able to put into that to infest is like here and here and here: okay now I only drawing one cell, because this is a predictive input right. So that's now.

G

The point was saying: okay, I went.

B

From an unpredicted input to a predicted input, I need to start pooling up here. That's the point of the whole thing right, and so what does.

F

B

It means I me how this guy has to be stable. Well, this guy is changing, and so, as long as these guys are very sparse in predicting this guy has to stay the same. That's the whole point and it has to learn. It has to learn that all these patterns are occurring down here are equal and this guy has to represent in the same way. So again, I did a lot of research trying to get the biological evidence on this, its basic Lucifer.

B

So what we're going to do and I'm not sure it matters if you get exactly why it or not, but what the simple way we're thinking about it now is: ok all right if I went from being unpredicted to predicted as long as I'm convicted I want to keep the same pattern and I want these cells up here to learn to recognize all the patterns down here that are predicted in sequences. Like you, cells appear at the state.

B

They have to learn to recognize all the sparse patterns and the melody down that this was a melody and I was going through the melody and I'm going through. All these, and they said I- could learn to recognize anything Amy estate that melody that clear, ok, so what we're literally going to do is we're going to say: ok.

B

Well, since I went from bursting to sparse I know we're going to pass these directly through right, I'm, just going to say, ok, I'm, going to leave these guys active and literally what we're going to do is when I look at any one of these cells- and it's got some dendrites out here- we're going to say: ok, there's not only do sparse pattern down here for every one of these stuff and we're going to just do the standard loading mechanism.

B

We do just try to learn the recognize these patterns from some subsampling of the Act ourselves down there. So all these cells that are bursting of all these columns are essentially representing this sequence down here and if I ever got back into a state where I was any one of these patterns on your existence. I would invoke this pattern up here and this.

G

B

Of stays the same until this guy is no longer predictable. That's that's the end goal out defriended and exactly the neural mechanism making this happen, and you know why does they have to do this and that's a little tricky and I don't get it I can't fully I can't prove that this could work so good work, I can't I, don't have enough evidence to say I'm, confident this is exactly what's going, but it solves the problem that we want to solve. It's pretty simple and solve the problem, achieves the results we want to achieve.

B

Basically, this is stable as one is. This is predictable and soon as this is going predictable, we go to the new state here and by the way it does something else really cool.

B

Think about vision, I'll leave you on this point, visions tripped up so many people when you actually look at something your eyes or ciccotti over and over and over all over the place patterns completely changing you're, not even aware right. Your perception is stable. Your perception of what you're seeing is stable. In reality, this bottom stuff is changing all over the place, but some of you goes nothing levels up in the hierarchy. Three or four levels of the hierarchy is stated, so we do example pulling now. The tricky thing is is I.

B

Show you an image and I, don't give you a chance to sakana all I can do. Is one image and you it's flashing from your eyes and you don't have a chance to get a second image. You can still do a lot of inference. You can still say what was that picture, even though you didn't have a chance to go ding ding, ding ding to predict, which is the normal.

B

So it turns out that if you, if you train a system like this and then you you throw it so you've trained it over the over Tsikata, you've trained it over sequences, and now you just flash it in front of it. What's going to happen, you're not going to get a sparse representation down here, you're going to get a complex. Are you going to bursting because it was unpredicted? There was no prior state and boom there. It is well.

B

What's going to happen well, even though this these cells appear learned a very sparse pattern down here, this input is a superset of that sparse pattern. So maybe it was in this cell and this cell and this cell I will still invoke these celibacy cells appear. We'll still see that pattern so like it's like, there's a whole bunch of pattern, superimposed on you because I'm not bursting these columns and it might it possibly might invoke some other interesting.

B

Is you but unlikely, and soon, as I did a little test on this already, and I think you could basically right you saying yes, so this is a really nice result. It says that, even though I learned by psychotic and interacting with the world at least envision is how they may be not anywhere else, but least envision after I've learned that way and I just flashing phonezoo, the learning that was occurred through ciccotti will actually invoke the right representation here.

B

It might evoke.

G

B

Several others at the same time, but very quickly filters out to say nope, that's the one that is so I can't expect you to to God that right now, I just want to point out. That's a nice result of this whole mechanism is that it does do spatial inference, ongoing vision, problems where, but it shows that it's a subset of the learning process of psychotic and motion, and that's that's exactly what we'd like to see. We want to be able to explain how it is.

B

You can do inference or the flash image, even though that's not normal vision. Okay, that's the end of my.

B

Done question Jeff yeah.

D

So on that last point it seems like in theory it should be possible to train a system to do that without any of the temporal cool line right and you don't limit yourself any specific impulse implementation in theory, there should be some way to do that. Do you do what to train a system simply to recognize an image without having to use a temporal process to training.

B

Well, I would say no question was: maybe people here you know seems like there ought to be a way of training, a system recognizing image without temple. Food well, I think you know, I take Google's shown that's possible. They use deep learning. Networks with you know, gazillion things into images. Somehow they got some basic who would occur. I would say that the brain that is it's an odd thing and at the brain, uses temper pulling that her question about in giant representations.

B

This is the mechanism by which I'm, very confident of what environment comes about in the brain is to temple pool. And could you do it some other way? You know, could you could you create a vision system that recognizes you know caption dogs, by training on a deep learning network with 16 million pictures problem? I guess you can I mean I, don't, but that's.

D

Not the way the brain dump- ok, yeah, but I guess. My question is more on. Think. If you just take the spatial pool or do you think, there's some reason that that can't be made to do that job? Well, I! Don't.

B

Need the special bulletin.

D

So it's not an archbishop moeller! Well, ok, but in any sort.

B

Of speak well, obviously, if I say that Google doing to keep owning that way, can you call that a spatial pool then? Yes, it can be done, but I don't think that would be the same as the kind of spatial cooling house talking right. I don't come, but what's.

A

Also question: how would you train any given Network, and so you know a network that only had that yea.

G

A

With labeled examples that show exists in the you know, yeah.

B

A

You can train with something maybe.

B

Like the biggest problem, I have but I've been this huge advocate for time in artificial intelligence and neural models in vegan and I think you will never ever get close to machine intelligence if you do not fully incorporate time into your thinking, and so you know, our CLA is part of a framework called hierarchical temporal memory and that framework we're now able to plug in spatial a sensory motor inference. We are now able to plug in motor generated behavior by the cortex.

B

These are all time based product behavior is the time is padded and and there's no way you could possibly do inference on auditory or somatosensory or any other thing. So is it possible?

B

Do some cool things in vision without time, yeah I'm sure you can, but that is not getting at the essence of machine intelligence that you're just going to get stuck in some local minima someplace, and so, if you just just think like all right could I have a system that doesn't have time to those that the has behavior I, don't think so I think that's impossible and can I have a system that does auditory inference without time? No, it's not possible. So vision.

B

Is this one corner case where the spatial patterns are rich enough, that you can sort of recognize the melody by one note you know I'm saying by that you see the analogy there and.

G

B

Know but that's not usually.

G

Possible in anything else, it's.

B

Just because of vision and vision is it's forty to fifty percent of the cortex. It's dedicated division because it sets a huge massive memory problem to solve that so I think that's a really red hair man. We don't you don't want to get down that path. So, when I talk to people who are the oiling people, I say what about time? Well, how are you going to act into your time, makes yeah we'll need any rate time you're right, but we're trying to do that.

B

We're stuck on it, whatever I think we know how to do that part and and and you need to integrate motor and you need attention all these things are time-based things so yeah you can do some stuff without time. But why bother you aren't voting machine telligent you gotta have time, that's! That's it yeah.

J

So little confused now! Well, what if your highlight the only way, this concept here, where you have any higher like this, was calm above this.

B

Is mission 1 CL, a layer? This is another steal and.

J

B

J

He notices what she knows.

B

J

Yeah he's a violent her.

G

B

For the regulator, brain I don't know I've ever thought it was an itch. So.

J

Is it so the fact that it it's got the same, these branches below it well,.

G

J

Just when that subset is active, it then says: hey these three things are active. I should be active, yes, isn't that special yeah.

B

But well, no, it's not sufficient.

B

Ok, I hope, I answered the question. The problem sometimes I'm, not good at hearing the question you said: isn't that special for one? What I decided is the mechanism? That's a spaceship or the temple pool are really the same. Maybe that's what you're saying it's really it's an extension that we merge them together. In some sense, it in a nice sense, you're right. In that sense, the saying: okay, this guy's littering to represent a bunch of patterns but notice. Those pants are no longer spatially.

B

Overland right is that feeding put two here and selecting the columns, but these cells going to respond to orthogonal patterns down here, not just ones that are close over London, so we're actually doing spatially and temporally in the same just if you were, if you're familiar with cl a theory what's going on in this cell here I'm saying the proximal dendrites: these are the ones close to the cell body or the spatial pooling ones.

B

Well, there are also going to be bullying basically they're, all the proximal ones, but I don't want to talk too much about it, this one because they don't know if I'm you even talk what you're talking about, but essentially we're extending the concept of spatial pulling with the proximal dendrites to recognizing multiple, unique on orthogonal patterns down here, and the only thing you have to do. We have to keep these cells active. We basically have to say okay when once the self becomes active.

B

As long as the comment below are predictive, I got to keep these guys out there as long.

G

B

Active they'll keep learning, there's long as I keep learning or learning about it anyway and I've done the math on this. You can interdigitate these multiple patterns, all these different condoms on the same kinetic segments, it's not going to get confused, so you don't have to have separate dendrites, for this is reckon, is one pad or the other time you just throw them all the other there. You know separate out those patterns as long as we're very sparse in the arbor, by definition the first course.

B

So, if your question is, isn't this like space a hole in it before I like it, but we've extended it to recognize orthogonal patterns as it, because it's filling the role of God from one region to the next? That's it's like spatial pulling in that regard, and it's determining which of the active columns. It's like spatial in that regard, but it's now low under just spatially. Similar patterns is now temporal predicted secret crackers or thank God okay and one last customer, like a not gonna end this you.

I

Got right so, if we're taking the region port right now we're taking it looking at the crowd booing, so we have the first pump triggering the second one. That's reading the third one, because it seemed that this ABC, like 18 ABC report, then when ccp thinks his aides good, trigger beam and see. That's that's! That's temporal bullying by the region, I think I, don't add my baby. She, okay, so hey zippers, calling all.

B

Right these are not. These are not. This was one pattern. This I was wrong. One boy confirm this.

I

Is one point I'm remember just.

B

Sparse activation with arms, but it said: there's no column a there's, a there's, a pattern, a which is this pattern and another time I'll have so may God, say: okay, I, get this column, this column and this column and B I get this column and this column in this column and see I get this column this column and this con. Something like that right. So my question.

I

Is so if you have a that was active first and I mean if you've seen many times a then B, then C, then inside the inside the region, for there will be several cooling that will cause be when it happens right. It's.

B

Not technically that's secret among Syrians, equus memories, yes, I'm, separate I, invited I've said this on the line, but unfortunately we use the term temple pooling in the white paper referred both sequence memory and typically- and it really should be separate, so the sequence memory and then there's temper point secrets: memory occurs within here: learning what's ABC and the attack will pull into the courage when I go up to here, which is saying group those together. My.

I

Question is so we're doing this inside layer, 4.

B

Yeah we're predicting for predicting day after a predicting say after being.

I

Ready, so my question is: if we're doing that this inside layer of work, we also take the insight that if the night inside layer, 3 gut is activated because a and B got active, the seat should get predicted by the layer, above it too. So if there is one representation for ABC well,.

B

I wouldn't know I would know to do that feedback.

I

Feedback like you cuckoo the dendrites actually in some way or exiting the axons of their.

B

Ego player 3 help repeat it for exactly that.

I

B

Ag you're absolutely right, and that is another talk. I really think I've made some progress on that how this guy feels got to this guy and what it's doing, why it's doing that and I'll do that. Another time, that's a lot of stuff to talk about, but you're right. It turns out that yeah, you know well not later I shoot I'll take about not letting through the way before. But if I go back to erase.

A

B

I risk, if I go back to there is there's really no feedback for money through that. Fourth, but what you do have is the following: this is what you really have you've got a region in the cortex. We just got layer three, four and five and six for this thing. You got another region which is hierarchy below it. Three or four five be another asian feeling like this. They have three four and five. What you.

G

B

Have also sticks go to there because from player six you see these cells that massively projector hear from later six. You see the cells that massively projected here. There's this big feedback path we going down- and this is a few third pathway- and this entire region tells this region. What to expect in this entire region tells this region want to inspect, and I think I understand that pretty well now, and that would be another talk.

I

About the biologic eat quickly, could you just tell us what's happening, then Droid actually exceeding the accent or how's. It literally.

B

Works like is our cells and layer, sex, all of them wats on that project, and they spread their actions right broadly overlay a one year, then cells in layer, 3 and five have what are called apical dendrites to go up into layer 1 in addition to the basement entrance. So when we model the remodel cells in the CLA, we just talking with a basil filter, that's the sequences, but this these other dendrites that come up here and what these guys are getting is feedback from this, and so what's actually happening on setting one sentence.

B

A stable pattern here invokes a union. It didn't folks doesn't make themself. Are it just it just be polite. Put some in a predictor stay depolarizes a.

B

Union of all STRs in the sequences that are represented by the state are represented by this data stick, so it would be like saying, I'm telling you to do a motor command, and this that says, do this motor command, and this is going to take all those later five cells and invoke all the possible motive. Sequences that could generate that behavior are all going to be default, lies at once and.

F

So it's like it! This.

B

Is that this is a true regarding representation says these are all the.

F

Things I know about how to achieve.

B

This and you're going to pick the right one out of it or you could think of like your I'm, expecting to see a gorilla. So these are. These are the these. Are the cells that should be active on all the different ways, you're going to see a girl with them.

I

B

A minute predictor state shows a union of all the things that could possibly be participating in this stable state. It's like all the notes in the melody or invoked at once, we're expecting to get those notes, and that's not a good example, but it gorillas a better one, but anyway, and and and so now these guys says. Okay, all I have to do. I've got a union of all these things now I have to do is take an input and that input will select where I am in that sequence and bias it.

B

So if I see a fuzzy pattern house they- oh it's the grill, because it looks a little bit like a gorilla and I'm biased to see a gorilla or I'm going to say, okay I'm in the kitchen and to get the drink I need to go. This way, don't be like that. So that's the basic idea. There's this feedback, which would vote communion, just the opposite tip of pulling it's a good way to end this up-tempo food is taking a series of pounds here, form a stable pattern.

B

You now check the stable pattern here and go to the union of the sequences that led to the stable pattern so that I now can take an input and uniquely select one of those elements and until I actually low road up there on I couldn't believe it automatic. But I didn't understand the back to her before so it is boot.

I

The whole column in predictive statement.

B

To individual selkies, the continents very individual, suffice to say yes,.

F

B

The point is, if I see this fuzzy input that could be interpreted a lot of different ways. I want to interpret it as the gorilla I will get the.

G

Immediate, that's the girl, even though.

B

It's just a fuzzy guerilla and I, wouldn't if I just saw it on its own without that, without that body.

G

B

I wouldn't know what it is, so it occupies all the cells and fire, but with the bias it says, the polarized cells fire and also you get bingum. That's a good rule. Ok, we're going to.