Numenta Numenta Research Meetings, 3 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sensorimotor Brainstorming (Part Two) - June 3, 2020

Description

In this short research meeting, Marcus raises some questions about Jeff’s brainstorming session on June 1.

Link to June 1 brainstorming session: https://www.youtube.com/watch?v=uQpX-MnAJqU

A

We're posting okay, so yeah on Monday, Jeff posted some slides and I was gonna, give like a two minute response, but but that was gonna provoke a longer conversation, and so this is like an informal informal follow-up, just acting as if Monday's meeting everyone did I know you don't have to because during that last meeting, I sat and drew what I would have written on it on a whiteboard on Monday, so I'll project that now I'm ready. This is I didn't use a physical labor board.

A

I used a electric whiteboard known as an iPad, just second I've been debating whether I should.

B

C

That yeah I mean I've been using that some in some of my meetings. It works quite well.

A

So yeah and with zoom, you can well okay, at least on a Mac. You can plug your iPod directly into your I know into your laptop and project. It like this as I'm doing so. This is not this. This. This doesn't meet my usual bar of thinking, deeply about a thing and presenting a finished thought.

A

This is what I would have said Monday after your after what you you said so, but just like saying back to you, you saw on Monday, you talked about flow inputs to cortex and and that being something that's kind of fundamental and part and like deep layers of cortex are may be recognizing flow inputs or representing flow inputs, and such and I just wanted to to kind of map this onto it's just I'm, just saying it back to you in in different language and and and yeah applying it to two different phases.

A

When we talk about cortex were often using something analogous to what I've drawn here. Is this the rat in the room? And that's that's: that is physical space? That's the that's yeah that that's kind of the baseline for all of this I.

A

Think, when your motivations for not having a motor input by required, is that that you want to be able to apply this circuit, this algorithm to more general spaces like conceptual spaces and and once you no longer require a motor, and that makes the system kind of more powerful and what it can represent or how it can build. Do you want me to.

B

You might add a couple more for that. It wasn't like I wanted to get rid of the motor input. It was the motor, but was always a very problematic thing for me. I always felt that a cortical column would have no way of interpreting an efference copy motor signal from someplace else. It wouldn't know what it is wouldn't know what it means it had to be learned. Somehow it was that it's not something you could know up front, and so that was a problem it's similarly the motor output or cortical column.

B

It couldn't know what it's supposed to represent, and so that had to be learned so the whole motor mapping, the efference copy motor output was always a problem for me because it was it was it had to be a secondary step after some other learning occurred, and then I think you're right point you're about to make your you just made was that the motor output is also something that's a little problematic to think about when you get the higher-level cortex and although we always talk about every cortical column in the cortex having a motor output, you know the person who really argued that to me was Mary Sherman, but he always qualified it saying well everywhere we looked now everywhere.

B

They we know that there's layer, 5 selves everywhere, but we don't really know that every like five self projects, subcortical ii, we do know that project to other places in the cortex. So the idea that some parts of the cortex may not actually not not be mapping to feel the motor things. That's another issue, so I'm, just adding on to what you said there.

A

So what I've drawn here is like yeah, the baseline space of a rat in a room or a sensor moving and space around an object, and there you have this there. There is a subtle difference between the two spaces I've drawn here. Isn't it maybe it's not that subtle! This I can't one with like this bird's face with the stretchy next legs birds being mapped into a physical space without one the directions of movement are there's no notion of egocentric. There's you move an X based in you, move in leg space, there's no notion of!

A

Is it in the space of the body or the or the or the.

B

A

Yeah so so, like I could have so I'm. The bottom row of this I draw I, draw the flow arrows and they correspond their relative to the body of the rat. It's maybe I should have used different, not a nonliving object on the right. It makes things a little confusing. This is. This is a conceptual space, so the the for the for the left column here I could not draw those arrows in the space itself.

A

If I'd drawn the arrows in the space itself, that would have been confusing because the arrows need to be egocentric yeah, but on the right column, the arrows could have been drawn in the space itself, thinking they're, just they're, just parallel to neck Lincoln, leg links, and so anyway the whole egocentric thing can confuse the conversation. Sometimes so that's that's how I how I drew it here I feel like it's not that important of a thing. It just really confuses the conversation, but there.

B

Is still a very important question which that's alluding to Monday, which is we have you have to figure out how to get from egocentric to allocentric space, eventually it has to curve. So that's an important point.

B

D

B

Massacre and I'm starting to talk about that on Monday when I was talking about perhaps or two different, the upper layers and lower layers and epilator modeling movements of an object moving from body and thinking that that problems.

A

So so I'm attracted to the problem space of of how do you learn these spaces these these spaces, like I've, drawn here, even if you don't have a motor in? Can you do it just by first of all the most extreme case? If you just have a pure sensory input that doesn't even have a flow component, it's just sensory input.

A

Can you learn the equivalent of a space like this and then what happens when you do add something like a slow input, that's sort of sort of helping the system learn, so so here's the second set of things I drew to work. So yeah, I literally wrote this like ten minutes ago and the handwriting's bad, but so more generally kind of a point of view here. One way you could choose to frame all this, not necessarily the right one is, is so assume your sense.

A

This has kind of been the kind of in the language of how I described this. If I were putting this in a more mathematical framework. So but it's really different words for the same thing assume the sensory inputs can be mapped to a low dimensional. Euclidean space or a low, dimensional manifold- and here this is the equivalent of like our maps or reference frames, and and so these sensory inputs they might be nearby on the map, but they might be very different from each other.

A

Here, I've drawn three sensory inputs, I didn't do anything to indicate it, but one and two might be very, very different from each other and one in three might be very similar to each other. So so, instead, what is the distance on this manifold? Do the district distance and the space one way? One way to do that.

A

This has been shown to work where you can, where you can sort of learn these spaces is that you just use time as a supervisory signal where the distances between inputs are proportional to the time between when you experience them and and of course you don't have to learn, you don't have to learn all pairings. You don't have to learn the timing between 1:00 and 3:00. It's more transitive you! You learn the timing between wine and the space around it and then in the space around it and it all just kind of works transitively.

A

So so some people even studied grid cells. With this question in mind of, are they actually learning temporal spaces? And it's just the fact that that these rats are moving around these box tripped rooms uniformly and that's what gives you baby it's nice grids, but but in the end, maybe it's all learning a Euclidean space like this and using timing, as as the Supervisory signals.

B

A

B

Are popping in my head as you're talking here? This is useful, but I've started to think about these spaces. You know in this I thought about these spaces. Oh you can think of it. Like you know, Cartesian coordinate or something like that, but I started thinking about differently. I've started think about the spaces were defined by movement. That is, they are movement, spaces which, which can be mapped on to physical things.

B

But if my, if my criminals to the cortical column are flow bits, it is in some sense it's those flow bits of random process them they. Basically, what I'm dividing up the space is, the different movement, primitive, primitive, so I, don't model I can go over the world. There's a model that of the part I move through I can't model any parts that I can't move through, and it's just a different way of thinking about it.

B

It's a way of thinking that the space themselves is a movement space united into a unites movement with with the space and with the sensory input. At the same time, I don't know if this being clear but I've moved away from thinking about I'm like oh. This is a direction in the space that's meaningful on its own right. You know it's only a meaningful direction. If the animal moves in that direction and whenever moves in that direction doesn't exist in some sense, I know: that's it's just a it's a thought, I wonder.

B

But what you just said reminded me: it's like because, oh I know I because movements are always temporally cohesive right. You know you don't randomly jump around places, move through space to get someplace and so there's always an implied when you're moving in a dimension in this space. You're, always timing, because animal has to before thing has to physically move in that direction or in higher cortex. It works you're, not physically moving, but in the lower sensory I.

B

Think I'm, just saying what you said: putting good words on it, yep well.

A

I just had these two points remaining and then I'll just open her up for general discussion, so yeah two thoughts, two reactions based on what I've drawn here and what we've talked about, having an actual flow input, if you can like just for free sort of, if I just give the system for free this, this SDR that activates, whenever you move, left and see our that activates. Whenever you move right or something along those lines, it makes this learning problem easier.

A

But I can flip this around and say well.

B

Can you explain why makes it easier what you think yeah.

A

Sure, if you had a set of cells that they're just like opaque cells, they're, like um movement, Direction cells, similar to head Direction cells, but their movement, Direction cells, then.

A

Using that, if you don't lose it, maybe it's easier and say if you don't have that learning how to map your sensory inputs and did just in a Euclidean space is a hard problem to solve. But once you have these flow inputs, you're, essentially receiving a label for the earth.

B

You say maybe flow is your time. Co adjacency is everything, that's your supervisory signal, but the flow itself is.

A

Let's see as you're mapping sensory inputs into this space.

A

There is ambiguous where to map them, but it's not as ambiguous once you have a flow input. I.

B

Agree with you, I was trying to scratch down below the surface and try to figure out what why exactly? Because you might have a different. You might have a different way of deriving that same insight, as I did. I did I just see if I could that's all I was asking I mean I, agree for statement, I'm. Just just wondering about your your. You know your uber thinking that led you that you know what was your visualization.

A

So if, if you, if you move to the right and you okay, no sorry let me say this different if, if a flow cell, activates and unbeknownst to you that flow cell represents I moved to the right, but it's just an opaque closed cell and then it happens again.

A

You know exactly where, in the map to map the second sensory input, this, like there's ambiguity on the first one, but not on the second one. Okay,.

B

So that that's a little bit just sort of like our whole sensory motor model, that we did in the columns paper that it's like. Okay, we have your sensory inputs coming in, but we can't really make sense of them unless we know what the movement commanded is that similar to that, did you yeah, I? Think.

A

So, okay, so final thought is now just taking that first bullet point and inverting it. If you can learn these spaces using just the temporal trick, then once you've learned a space it can be. It can give you these flow labels. It can give you the ability to detect flow. Oh well,.

B

I thought you use flow to detect the build on this space, but now you're, saying that building space I can do in.

A

Principle, you can build it without flow if you just use the temporal adjacency trick, if you just but.

B

That seems almost like a comment orally impossible problem. You know you're basically saying oh I can just watch how these different bits change over time and from them. I have enough information to deduce underlying structure which may be true, but the number of ways that it just it may be. Computationally tractable forgot to do that, but flow gives you. This structure flow gives you these like dimensions of the space and then all sudden you can say well, you know, I can now say that I'm moving right. Therefore, I can eliminate all these other possibilities.

B

It could happen. So what.

A

B

You disagree with that that it's like almost computationally impossible well.

A

No well other other people would respond to this with that that, if they would see, hebbian learning is similar to principal component analysis and principal component analysis does solve this problem, basically, where, if you have cells that are that are, let me try to say this precisely, not not not over complicated language. So if you have cells that are I'm, sorry I'm not prepared for this heavy and learning as like principal components, principal components of of location are a grid like.

A

Six months ago, I would have done better at this.

A

B

C

Question about some of this. Well, it's a small question, but in the case of images you know how. How would you incorporate things like different distances, because and or maybe speed of movement, and things like that? That's those are seems to be additional dimensions that somehow have to be incorporated yeah until recovering.

A

I agree, but I think that that can be accounted for. I think that you can I agree that this this is kind of a cartoon version where you it's almost like all all things move equally, it's like the speed is constant. If I hold the speed constant, this works once speed can change that it becomes more complicated at the same time in the ended, averages out and works out, and you.

B

Know you're amazing, bead of movement, the plot, the actual velocity yeah. If.

A

You use time as the Supervisory signal. Then it starts to become kind of strange because sometimes you move quickly. Sometimes you know.

B

Yes, you know I've been doing a lot of work, think deeply trying to think deeply about the whole. You know good cell path, integration problem, trying to get it's like a deep understanding or all the different models and how they relate to each other, and you know, and one of the things that you get of course from that at least all the variations were the oscillatory interference models. Is you get the sense that the path integration takes, or it automatically takes account of the speed that best to say that?

B

But you know if I had many column, the representative direction the one to move in a particular direction: the scalar activation of those cells in the mini column. We represent the velocity in that direction and that would change the Ocelot, this control oscillator, and that would basically then the whole space.

B

It abstracts out the speed components that you do path, integration correctly based on different speeds, so I just turned that that's one of the nice things about the whole oscillatory interference models that it it automatically all this flavors in that it takes care of that velocity issue. I just I'm not sure that she dressing what you're just talking about. But at this it was related to look like much be in velocity.

A

One alternate responses that actually it is that if, if speed change it okay, you move quickly, you move faster in some places, then you move in other places and and that's actually going to result in your grid cell map being more.

A

If you use this learning rule your grid cell map, where you can.

B

Which learning.

A

Rule yeah the using time, is a supervisory signal and and see what I mention that that once you bring in speed, it gets more complicated, but but the implication here would be if there are places that you move faster than others. On average, the grid, the Euclidean space or whatever the sensory inputs, are going to be mapped into the Euclidean space and kind of a warped way where distance spatial distance is not constant and yeah.

B

So actually I'm just saying, though, all the Greeks are module models involve oscillatory interference solve that problem. You don't have that issue.

A

I mean I think the same is true with attractor models. Oh am.

B

I guess I I guess I've just given up an attractive.

B

The boy get into by the way the more I'm convinced that the the oscillatory interference modulus gets such powerful stuff. That you know, that's why I was excited about the flooring presenting hybrid models. You know it just seems anyway. My current thinking is I'm I'm, not my.

C

Customer would be thrilled to hear you say that.

A

Another thing, I was the thing I was trying to say with the last with these two bullet points was there's sort of a bootstrap. You can think of this as a bootstrapping trouble more opportunity where, where that both processes can help each other out where the flow helps you learn the map. The map helps you learn. The flow I think.

B

That's all right, I think it's good but I to me. Would you disagree? Then? If I say you know, but it's a bootstrapping process, but I can really get started with flow. I mean I, mean flow bits, our basis to build a model and and then maybe you can go back and forth as you go. But the nice thing about the flow idea is that it applied to anything. It just applies to any neural tissue.

B

You know a column can be looking at another region of the cortex and it can apply the same principles and it should work it just felt like it was a very fundamental idea, so I I'm, not sure I you'd, agree that you could get bootstrapped without the flow and you get the flow. Think you'd get your going and then maybe you could speed things up by reversing your reversing yeah.

A

I guess a motivation here was that was like moving okay with sensory input with our visual input. We certainly have these like fairly unambiguous flowing. That's especially if you have the whole visual field, you can imagine, that's might not be true in conceptual spaces. I.

B

Don't know, maybe there's no I, it's interesting. That's a really ancient question. I haven't spend enough time thinking about it, yet I'm still just trying figure working the sensory spaces, but what you know it's an interesting couple of clues. You've always had this challenge to figure out. What's going on with you know these layer 5 circuit through the family, so we, you know they were say: five cells and every time, every region project to the lamech relay cells which project to the next hierarchical region and that we've always struggled trying to understand been al sub.

B

You know so that's like a motor commando, let v if it does projects up courtly and it's a motor command, but I'm going to argue perhaps some places higher in the cortex. It doesn't project importantly, but today I'm getting it there's some structural clues as to how to think about this in higher-level regions and and so now, I think about now. Thinking about the Islamic relay cell and I'm, primarily thinking about them as these are the flow become. Should you know these flow goats are going to this lemak relay shelter?

B

Why that's not just century bits but flow bit, and and what transformation would I do on these globe? It's going into the.

B

Yeah to the next region- and you know, motor commands are always gonna anyway, I'm sort of just stuttering here again, I'm, sorry and I think there's some clues that we can use to figure out what's going to higher-level regions and I haven't really spent enough time on it, but I have a sense that we'll be able to figure it out if we, even even, if speaking, to start thinking about it from a structural point of view like what we know about these connections and whether they look like what would a higher-level column be observing and a lower-level cop.

B

What kind of outputs from the lower-level column have and could we interpret those in terms of flow and non flow information anyway? I think this is a good. What I'm, seeing from you're presenting your Marcus is that you're you're interested you're, not you're, not covering this idea, you're sort of adding some additional flavors on top of it, correct, which I think is nice and III, would I want to just every emphasize again to me that there's there's several there's several problems that we have to address here.

B

That I'm still trying to get my head around one is the transition from egocentric allocentric spaces that has to occur, and then there's a question: does it occur within each region, each column or does it occur between them? The second time we have to do the compositionality thing, which is artists, placement idea displacement so so that same question goes.

B

There is a happening within a column or between columns and then the last thing is the sort of the state of an object which is sort of we haven't seen space of the object, but it's different states. We talked about that. These are. These are three major themes.

B

We've talked about over the years and and I'm trying to figure out how it is that a column can do those things or two columns connected hierarchically can do those things, and you know maybe I, don't know the right one I'm trying to solve all at the same time and I've done, but that to me is like the really hard enough: the crack here and I'm, hoping that this idea of a mini column, movement, space idea and upper and lower layers, and also the the the attended part being the upper layer and not attendant part being the lower layer, could could help figure out the answer to those questions I'm just dating, where my mind is at right now.

B

The problems I think if we're gonna work on this. Those are the front you're going to think about this Marcus and come up with some brilliant ideas. I'm trying to play it a problem so think about well.

D

Marcus, what you're trying to present do you need a metric space or this or would a topological mapping be sufficient.

A

So the metric space- that's that's created here by those simplest learning rules as as average adjacency and time or average temporal distance between points right.

D

I, don't know what you.

A

Mean by topological, not in.

D

In other words, when we, when we think about a metric space, we think of somewhat absolute positioning in a sense that I could you know, do as Jeff was saying trying to solve a path, integration problem, or something like that, if you're trying to show that these flows, you know lead you from one place to another, but not necessarily, you know in a way that if you did a circle around the thing you wouldn't you know integrate to your starting position for what you were trying to show.

D

Do you actually need a hard metric space or would a topological space say buy head off in this direction? I will connect with this thing if I head off on this direction, I have in this this way, as long as there's a rough idea that there's this kind of path through there, but not necessarily a very warped path. You were talking about how speed was a difference in speed was gonna. You know corrupt your vision of the thing so I'm just wondering. If there's a you know a less.

B

Like you're, saying Kevin like Oh, am I going this direction. I'll get to the library. I have no idea how long something.

D

Like this direction,.

B

D

Like that, but in in solving it in the context of the space they use right, but.

B

If you're doing it, if you're doing that, problem like how do I get to the library from where I am, if I've never passed that, if I've never gone that direction before but I've never observed and actually executed, that particular from this location to that location, then I argue you have to have a metric space to do that, you can't there is no other way of doing it. There's no other way of saying: what's the distance and direction from here to point A to point B if I've never gone there, so.

D

I can make spaces. He was talking about. That's not necessarily necessary. If he's talking about ahead yeah.

B

I'm arguing that that's an essential characteristic of all cortex and therefore that's my arguments like having other spaces that don't have the property to me or that seemed like a dead end. Well,.

D

It would seem that if you're acquiring everything to operate that way, then you just allow the possibility that you could have two mechanisms, one of which tells you you know the topological in action and the other one operating in Jason's today, okay, if I'm, actually gonna move my body through something I will use both of them together. You know so I'm just.

B

Well, I guess I guess it seems to me the the metric space solves all the problems we need to solve, and so, if you want to have a different type of spell use, it might exist, but I don't know why I would need it. It doesn't seem necessary. I have an attic space like I can solve every problem. I'm.

D

Not sure what a metric space and although magic.

B

To me, it's got two things you can take to two arbitrary points in the space and know the direction and distance between them.

D

So what about the case I just mentioned, which case well.

B

I, don't think you know the idea that I would know the direction, not the distance. It's not useful. I can't factory space. Well, we don't think well factory spaces while representing the cortex at all, and so we don't. We don't do anything that this we don't. That's, not a cortical.

B

You know in humans. It's really poor.

D

C

Is the subcortical thing yeah the cortex? You know, at least in most mammals like.

B

You think you can associate a smell with the location, but you don't navigate by smell. You.

D

B

C

B

Way should I move my nose to get there. It's like no.

D

If you're, trying to like and you know, smells together, there's some space in which you're trying to say.

B

Anyway, odor is not represented; well, it's mostly something cortical, it's very special organs for it, and so we just generally don't think about it. Yeah in these context it doesn't, it doesn't fit the sort of almost all the things we think about a touch and hearing and vision.

B

C

We should probably stop soon yeah.

B

I, just just interesting matter of protocol I'm working on this as much as I can, until I get back from the book again. I'm gonna continue working this I'm totally happy doing it. I don't know other people. How many other people want to participate in this and have these discussions more ongoing basis? I don't know, I saw they don't want distraction, they work company, so I'm. Just simply not the answer.

B

This question right now, but if you, if anyone has a thoughts about that like we have a few people get together different separately or something you want to really talk about these things, then we could do that or we can do as a group and just do it yeah. We.

C

Can do it in this research meetings? Okay,.

B

C

You feel like it's, it's not helping you, then we.

B

Know yeah, then we should do it, but you know I, you know, I, don't you know, I could die. I could probably dominate every research. We.

C

Can deal with that, one that happens all.

B

Right, let's assume we'll keep doing that I do want to go over the more a pattern paper. Sometime I, don't think the right now paper, but it's fascinating to read, and it really taught me a lot about it gave.

A

B

Much deeper understanding of what's going on and all these oscillatory inference months. So that's why it's fast so sometime we can review that.

C

Ok, I'm going to start either recording will.