Numenta Numenta Journal Club, 27 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Insights on Sensorimotor Object Modeling from “Animate Vision” by Dana Ballard - 27 September, 2021

Description

Marcus Lewis reviews a few papers from Dana Ballard and highlights some insights related to object modeling and reference frames in the Thousand Brains Theory.

Marcus first gives an overview of what “animate vision” is, as outlined in Ballard’s papers, and defines optic flow. Marcus then makes a case for using a world-centric, viewer-oriented location relative to a fixation point to represent objects and depth.

In the second part of his presentation, he looks at Numenta’s previous sensorimotor research (where the motor command is being received by the system) and Ballard’s sensorimotor “animate vision” system (where the motor command is being generated by the system) for objecting modeling. He evaluates whether the two sensorimotor frameworks will lead to different object modeling solutions and discusses the opportunities that could stem from Ballard’s framework.

Papers by Dana Ballard:
➤ “Animate Vision” (1990): https://www.sciencedirect.com/science/article/abs/pii/0004370291900804
➤ “Eye Fixation and Early Vision: Kinetic Depth” (1988): https://ieeexplore.ieee.org/document/590033
➤ “Reference Frames for Animate Vision” (1989): https://www.ijcai.org/Proceedings/89-2/Papers/124.pdf
➤ “Principles of Animate Vision” (1992): https://www.sciencedirect.com/science/article/abs/pii/104996609290081D
➤ “Deictic Codes for the Embodiment of Cognition” (1997): https://www.cs.utexas.edu/~dana/bbs.pdf

Papers by Numenta:
➤ “Locations in the Neocortex: A Theory of Sensorimotor Object Recognition Using Cortical Grid Cells”: https://www.frontiersin.org/articles/10.3389/fncir.2019.00022/full
➤ “A Framework for Intelligence and Cortical Function Based on Grid Cells in the Neocortex”: https://www.frontiersin.org/articles/10.3389/fncir.2018.00121/full
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.

Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://numenta.com/news-digest/

Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter

Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta

Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/

Our Website:
https://numenta.com/

A

Cool yeah I'll get started uh so here I am doing. I wouldn't call it a journal club so much as um it's somewhat like a journal club, but I'll just skip ahead to the next slide, to show you the list of papers that I've been looking at and trying to glean some insights from so and in this presentation, I'll bring forward just a couple of those insights.

A

It's likely that moving forward I'll present more ideas from these papers, or at least that they'll influence my thinking. But there's there's some stuff here that I thought was relevant to this to this group. Some useful things to talk about um so dana ballard uh he's been involved with tons of stuff early on. He did he brought forward like the huff transform to generalize it to other things which is related to reference frames, uh and then, in the um you know, the late 80s through the 90s and beyond.

A

um He has done a lot of work on principles of vision systems where the eyes are moving, where you're fixating, on different things uh and or and just in embodied vision in general, which just heavily overlaps with what what we're focused on and just over the years.

A

He had a lot of insights and I'm not going to try to give like a long presentation on all of those there's a couple things that I thought were worth bringing forward here, and uh so so my agenda here is um two separate points that I thought were particularly relevant to us and I could have done more, but this is going to be enough for for one presentation, the first one, the first one here I actually didn't grasp this until last night.

A

I was preparing this presentation and then suddenly I rewrote the presentation, because this first point clicked- and I realized that's- that's what I should focus on um so uh first, I'm gonna. What I think I'm gonna do here is background. uh He dana ballard in these papers talks a lot about world-centered reference frames or object-centered reference frames, and it took me a while to understand what he was saying. I had a mental barrier because we have so much experience with these concepts. I it's like.

A

I had all this baggage that kept me from seeing what he was saying. I was trying to figure out what he was saying in terms of what I already knew and it just blocked me for a while.

A

So I want to remove that block from anyone else here, because I suspect you would fall into some similar thing uh and then second is a is a related but different topic involving how does sensory motor inference and learning change once the system is generating its own behavior we've focused on models where it receives a motor command and it just kind of uh generalizes over or rather it can handle any situation with any motor commands.

A

Does the problem space change once you uh are generating your own actions and does that make the modeling task a little easier? Maybe have we missed any of those opportunities?

A

So these are two separate topics that I thought were worth bringing forward, at least on this first pass of marcus.

B

He is he almost a pure computer science guy, or does he also relate this to neuroscience? Does he.

A

He definitely relates it to neuroscience. uh I would say his his his original background is more computer science, but it's very much I mean in these papers he'll he make. He has theories about um here's. What visual cortex is doing. Here's what how the basal ganglia fits in so.

B

He does connect. Are these? Are these detailed ideas? Are they sort of uh general ideas I mean? Does it get down to like you know, neurons and systems, I mean I can see it. Maybe I'll present it I'm just curious how deep he goes.

A

And these it's it's more, it's not down to the level of neurons, but in some of his other work he does do that. He does.

C

Get down to the level of neurons, so it's like he.

A

He he plays between the levels, uh so yeah, um and also I mean like right. After all of this work, he was also involved with predictive coding, uh rao and ballard, the classic.

C

A

Coding is also him, so he's he's touched all of these things.

A

uh Okay, so I'll go ahead and get started with the first part, um first of all define so defining the term animate vision um that this is uh as I as I showed like the picture of the papers before this is a this. This shows up in a lot of these titles here that animate vision was the original work. It's built on some of this, some of their previous works, the principles of animate vision, reference, transformative vision, and then this this codes thing that this terminology, I don't think, really stuck, but the core ideas.

A

Do you? Don't I'm not going to define dec dick here, uh so animate vision is, as I said before, it's.

A

Framing, how should a vision system work, given that it has like a fovea? It's a padding, it's moving the eyes constantly multiple times per second, what are the principles of such a system and um and the the key thing that that he always emphasizes is that.

D

A

He calls gaze control is central to the system. uh The fact that this eye is always moving and has a few like core capabilities, um as is like a key design constraint or a key thing that yeah impacts the design.

A

um So, um first of all, um having gaze control being able to move the eye, uh it makes it where you can center the retina on features. It makes it where um it makes it where the vision problem no longer requires being able requires that you can handle a feature at any any translation on the retina. uh Rather, you can move the retina, and this can make the problem.

A

This can make the task of the visual cortex easier.

C

We're familiar with that, but but that is that that's a thing we.

A

Talk about sometimes that um that when you say cod to a thing, um we can trust that the eye's gonna succub directly to it it's gonna, it's gonna, um it's gonna find the distinctive feature and land on it and and in a canonical way. uh We've talked about that a little bit, but we haven't incorporated into the models really, but we've kind of assumed it a little bit.

B

You know one of the interesting things here, though, is that sometimes, of course the cortex is in control and says okay. This is what I want to fixate on, but other times it's not. Somebody else moves the eye, and so the cortex has to deal with that. So it um I'm just pointing out that it. We can't always assume that the you know that the system- that's modeling, the input is actually making the movement.

A

Are you talking about, for example, the video game scenario.

B

I'm just talking in general that you know the eye control is, is uh is controlled by the uh supercaliculus primarily and it can it can, on its own, generate the cods or tracking movements and so on, uh mostly you know, largely does and the cortex finds out about it or, but we know that the cortex can also control the superior colliculus. So I'm just saying that we can't it's both. I'm not disagreeing with any of this. I'm just saying that we have to.

B

We have to understand both sides, because there were times the eyes move and the cortex didn't move and that cortex just gets to say, hey we're moving, you know, and it has to do with that as well, so we can't we have to handle both situations.

A

I see yeah, I guess it depends how you define what this black box is. If this black box is v1, then what you're saying makes total sense. If the black box is the visual system as a whole, including the superior colliculus, then.

B

Well, yeah, I'm just saying yeah I mean we do know that that's that happens so sometimes again, yeah I. If the black box includes it's pretty colliculus, then the cortex itself will have to deal with somebody else moving your eyes and if this and if the cortex is in control and says up you move over here and then and that's all I just it's there's no other point just let me clear that it's not one of the others, both.

A

C

Second part.

A

Of gaze control that he that he makes sure to focus on is tracking a moving object. Keeping the phobia centered on a thing that's moving around also makes the vision um task much easier. It's a it's a nice thing to keep in mind again. This isn't new for us. uh But for me the the third point is: it was subtle here and I didn't it. Wasn't it literally wasn't until last night that I um realized the importance of this or realized that ballard believes it's important. Honestly, I'm still digesting it.

A

I think it's important, but I'm still digesting. It was until last night that I grasped it.

A

So as you move your head and body as as your body moves um a a system, a vision system with gaze control can remain fixated on a point, so you can continue looking at a certain point on the coffee cup, while you move or while, while you walk through a room, you can fixate on a particular part of doorway and even though your body is moving, it's not like your fixation point is. Trans is translating directly with your movement.

A

You remain fixated on a point while your body moves- and this is just a scenario that I hadn't um focused on and it might be a really important one for understanding.

B

It's funny. I always thought that we could ignore this. Maybe not. I thought we could ignore this and I say well, you know sometimes look like I'm walking around a room. I can stay fixated on the point and other times I don't right.

B

So I always felt like fixation um was one thing and then fixing a fixation on a new point is another thing and when I fixated on a new point, I may have moved my body or maybe I just moved my eyes, but I never thought that the it never occurred to me and I'm not sure he's going to make the point here that that fixating, while moving, is actually a it's an independent sort of method of learning or something like that. Maybe you're going to apply that I always felt like.

B

Oh okay, if I understand fixation while I'm standing still, I can understand it's the same as when I'm when I'm moving. I don't really think about it differently, um but maybe you're going to suggest that that actually fixation and movement at the same time is an important component.

A

Yeah yeah, that's, and that this this was new to me as well like this was the barrier that I I realized like. I said I haven't fully digested this yet, but at the very least, what I'm doing right now is making it where you can look at his papers and not be confused.

B

C

A

Be a very important idea. I I haven't figured that out yet.

B

I think one day it jumps out right away, is I'm fixing a point and I'm doing it right now and I'm moving my head? What will happen is the things I'm not fixating will have flow more flow and, and so that might be able to give your figure ground separation problems is. That is that, where you're going with.

A

That that's that's involved. That's uh you're correct to bring up flow. That is where I'm going with this figure.

E

A

Does come up, but I'm not going to talk about that here. That is a point he makes as well, though. Yes.

D

So I have a question there: if, if you're moving along, don't you tend to fixate on the part of the scene? That is not going to be blurred, I mean you're, not fixating on you know stuff, where that's moving very quickly, because it's going to be blurred, so you tend now you're walking forward. You usually pick a point. That's going to be relatively stable so that you're not getting blurred vision, but I.

B

Think if you're fixating kevin the point of fixing, it won't be blurred the point you're fixating, that most, who could be slowly rotating, but it won't be the point by fixing. I think what marcus is saying is if I'm fixating on a light, switch right now and I'm walking by that light, switch stays in the center of my vision. It doesn't it slowly rotates, but it's so it wouldn't be.

B

Moving by the point is by fixating you're you're, maintaining it um the only it would just be in any case it could slowly rotate or, if I'm walking towards something it might be slowly getting larger, um but it wouldn't be blurry. In that sense, sorry.

D

That's that's that's when you're you're fixating it so I mean it might distort because you're moving past it. But it's still, you would have clear vision. You know if you're rotating your eye to to maintain it, but you were giving the case of as you're moving. You know I think uh walking along things are moving and I at least I tend not to concentrate on the things that are blurry. I tend to concentrate yeah.

B

It wouldn't be reports of the vision, it wouldn't be blurry, that's the point. uh The point is you're fixating on it. Therefore, it's not blurry.

D

Yeah but I'd be fixing on the distance rather than you know, moving quickly.

B

It doesn't matter if it's moving past quickly and you're fixing on it. It's going to stay in your center of your region. It won't be blurry. Well, that's! Okay! That's.

D

Okay, no you're you're, making the case that you're making the case that sometimes things are moving and the brain's got to deal with it, and I I was I'm trying to make the point that you're always moving from one stable vision point to another and your choice of that might be to continue upon how you're moving I mean you can choose to say.

D

I want to fixate on the scene because I'm moving along the side, but I'm still fixating on one thing trying to center it in if I'm just walking along ignoring everything else in the room, I'm probably looking ahead at some stable point. Well, okay,.

B

So I think the station is, I think, there's a separate. We should really show that market through this presentation, but there's a separate question about what you would what's useful to fixate on um that's separate, but I think what I'm just reading the words marcus wrote down here.

B

The point is as you're moving around everything in your visual field is changing, there's nothing, that's stable and, and even the thing that's you're walking straight towards expanding, and so but his point is that as you're moving along, you can fixate on any point and the body will automatically track it. In some sense. I think that's where marcus is yeah.

D

We're only talking about translation here, we're not talking about the other transformation, so.

B

Anyway, when you're fixating about something by definition, it won't be blurry. That's the whole point of it. um Yeah.

A

And I think yeah the state, the only statement I'm making so I mean we say- let's say we said three times per second, um the the mental shift. For me. I was realizing as you're walking as you're walking or moving your head or anything. Let's say you still just continue cicada three times a second, um but as you're moving whatever whatever it is, you're fixed on fixated on for those 300 milliseconds. You stay fixated on it and then.

B

A

To something else, you.

B

Know it's usually I I'm I'm encouraging you to pick this up marcus when you, when a and when something moves by your eyes, will do a smooth pursuit. They will they no longer second, but they can track it and move smoothly, and during that smooth pursuit, I don't believe the eyes are doing fixations. I think they're just actually just being fixated and there's no there's no extra psychotic going on as you're doing smooth pursuit.

B

I'm wondering if I think that's right, I'm not certain, but I think that's right, um I'm wondering if I, if I'm walking past my light, switch and I'm looking at it. I wonder if, during that time it's there is no extra psychotic. I wonder if it's just if you're saying that it's not even three times a second, it's maybe just it's like a smooth pursuit. It's like it just says. I believe.

C

Both of those both of those are the same in that sense that you yeah, if you're as you're moving. If you stay, if you force yourself to stay fixated on the light, switch you're not going to be doing the cause, it'll be a smooth movement of the eye. So.

B

Those are really the same thing that, whether I'm moving in the light switches, uh I'm fixing the license or the bird, is flying by I'm doing a smooth pursuit. You're saying it's really the same sort of process, then.

C

I believe it's both smooth in that case yeah, interesting.

C

E

Have a question about the oh sorry.

C

Vivian go ahead.

E

Yeah, I think it's as far as I know, an open question of definition because in classical lab experiments this just doesn't come up and when I was doing eye tracking in vr, it was the first time that you had to distinguish between fixating on an object while walking and smooth pursuit, and it was back, then I couldn't find anything in the literature like that, defines it exactly because if you do eye tracking experiments in front of a screen that just never happens, I don't know if I know someone to find a difference between it.

C

Yeah I mean there's tracking smooth pursuit. That's been studied quite a bit in the psychophysical literature.

C

So there's quite a bit of evidence for that. I think um I.

A

Have a question on the first.

C

One, it says the system can center the retina on a feature. It does not need to handle arbitrary translations of the feature on the retina. The second sentence doesn't seem like a logical conclusion of the first. The second part of the paragraph, because I mean the system is not going to center every single feature on every single object right. It still needs to handle translations of stuff.

A

I guess what I'm saying is: when you're looking at a coffee cup, um you don't have to your vision system doesn't have to be able to handle every single single cod point on that coffee cup, uh like your vision system, can look at it and uh figure out how to figure out a better way to look at it.

A

That's the statement I'm making here. Is you don't? I guess I'm distinguishing it from like a passive image. Processing thing where, like a convolutional neural network, has to handle every single translation of a coffee cup. I'm saying that there. That is the problem that is being made easier here, but.

C

Still has to handle it because you can't possibly suck out to every single feature on every object, so you still need to handle translations. It's just that. You can make it a little bit easier because you can stack on a few of them.

B

I think he's making a separate point here. I I I maybe I correct me wrong, but in like some vision systems, computer vision system, they just scan over the entire image, but in the biological system.

B

That's not what happens and the system sends a fixate on points of interest that are likely to be meaningful um and is that what you're saying marcus.

A

B

Yeah, if you, if you remember the image that I that I stole and put into on intelligence, you know woman's face and you can see where the the cods and fixations guys. We all know, we've seen this it's you know they pick sites and eyes and nose and ears back and forth back and forth, and occasionally they'll go someplace else, but not very often so um so.

D

It's not random.

B

I guess it's what you're trying to say right.

D

Yeah, I would say if you can, basically, if the set of features fits within the foveal cone of vision, it doesn't need to translate within there and do that as a gestalt and then move away from that. Otherwise speed reading wouldn't work. So uh so I think when you I I take it literally. What you're saying is that basically, you can center on the center of something, but all the features within a certain radius of that center are processable without requiring detailed attention within there.

D

Once you go outside of that scope, then you've got a cicada. You know to get. You know that would be the natural thing to get more clarity.

B

I think that's true, but I don't know that's what marcus is saying here. I think I make markets just making a very simple point.

A

B

A

The points of interest you can move to the points of interest and that that's that's core to the division system. That's coordinated.

A

C

But it still does need to handle translations of features in general. It's not like you can avoid the translation in variance problem. Well,.

A

I'm saying it handles it by being able to sakad.

C

A

But if you look at the coffee.

C

Cup numenta coffee cup, there's probably hundreds of features on there, we're not going to translate we're not going to succumb to every single feature on that object.

A

I think this is a definition of features thing. Maybe I could have described this in terms of points of interest.

B

I I think we can just say the system tries to, for one reason or another doesn't randomly scan over object. This is well known and it tries to find bring into the photo yeah that.

C

Part, I totally agree with, I think, that's all. I think.

A

B

It's saying here I think, that's all you're trying to say at this point.

A

Yeah, that's what that's what I mean to say. I.

B

Mean and if you know, if you look at how we should hide over face, occasionally the eyes would go to some random place in the hair or outside you know, and we just have to assume the system can deal with that the system says well, some not all fixation points are very useful and if it's not that useful, that's not going to affect the learning much, but learning can be much better. If you pick more useful points to fixate on.

C

Right and you can also recognize things with flash inference without any psychology.

B

Yeah and we and we for the first time, really came up with how that works right, so um yeah yeah, okay, I I think we should go on this is the one. These are interesting, but minor points. I believe.

A

Yeah I'll go on the next slide by the way I should have mentioned at the beginning of this. The reason I got into these papers is because vivian recommended these and it was a very good recommendation, so I'll move on so a thing that is common in our work and in ballard's work is this proposal that some kind of world-centered coordinates are going to be necessary in early visual processing, uh but the a distinction I want to make here.

A

So this is common to our work and enemy vision, but the core motivation for these two things are totally different. This was the thing that that I didn't understand till last night. So in our work I mean our motivation, for these world-centered coordinates or object. Center coordinates is that it's better for learning models of objects, which is an old idea, uh as we've acknowledged it's just I I a lot of people have this same mental model that, like the reason for world-centered coordinates, is because it's better for models of objects mar wrote about it.

A

Hinton wrote about it back in the 70s 80s.

A

What took me a while to grasp is ballard's coming at this with a different motivation, so optic flow and um and what happens with optic flow when you can fixate, um they make the case that this core computation of optic flow well fixating uh that the world-centered coordinates or or coordinates based on. Your fixation point are the more logical thing to use. It makes the computation of optic flow itself much easier when you use world-centered coordinates where a world here is defined as the point you're fixated on.

B

And I actually prefer.

A

Is important but.

B

I have a question about this, so sure so there's two things that could be happening here when you talk about optic flow, one is you're you're, actually moving your head like you're walking around, but we're just talking about a moment ago. Yeah, the other one is my head is fixed and all I'm doing is moving my eyes and you'd still have some sort of optic flow there, but I'm not sure that you're talking about that process, yeah you're talking about when I'm actually moving like my walking around looking at something.

A

B

So so that's an interesting idea, but but we have to bear in mind that there's a hell of a lot of learning that can happen without moving your head. I give the example all the time you stand in the corner of a room and you're not moving your head, you're just moving your eyes and you can build a model of the world that room very well very good model and so that which at least tells us that this this idea of optic flow as you're referring to it here, isn't required for that meth.

B

That scenario um I just want to make that point.

A

Sure we're we're really good at uh inferring depth through other means, then, through moving our heads.

B

Yeah, so why so, I don't want to jump ahead here. Maybe if it's just a matter of an alternate way of perceiving depth. Well, then it's a useful tool, but it's certainly not the only tool and and and the system's not going to be 100 reliant on right.

C

A

Okay, right, uh it's so yeah this will. This will be an open question after this presentation, whether whether this is core or whether um this optic flow based on fixation and yeah object flow with fixation essential to how the system gets trained in the first place or if um or if it's just one of the tools at that. I don't know.

B

Well, at least, we know that I just gave you a single scenario there, where we know that um the you know you're standing in the corner of the room, that we can do very sophisticated modeling without it. So it can't be.

A

But at that point you've been so trained to be able to detect depth. uh Maybe maybe it was because you've moved in around in the world so much, but.

B

But I don't know, but at the moment maybe, but from after that fact, I do not need to move anymore and I can just move my eyes and I mean teaching. How do I know depth? I can even do it with one eye closed. I don't even have to do it, it's better with two eyes, um but I can do with one eyes closed and so there's a whole bunch of tricks we use to detect depth yep. um Obviously I was eating an interesting paper about that. Just last week,.

A

I guess a lot of those are like heuristics, whereas this is just like a core.

B

Yeah well, fine, but it works. The heuristics work.

A

The previous this is.

B

um Okay, maybe yeah all right, but once you train it still work. You know.

A

Yeah I'll I'll move on I'm going to show this figure more some details of it, uh but first.

B

Did you make that figure.

A

B

I saw the coffee cup there and said: well that must be our coffee.

A

No, that's the thing like that. That's the thing I won't talk about watching this presentation, but uh ballard makes a lot of statements about coffee cups.

B

Oh my gosh, I didn't know that totally unrelated.

A

So I'm gonna, I'm gonna dive into some of the nuances here. You'll notice that there's like there often we're talking about this or something analogous to the c prime reference frame or the c reference frame object center coordinates where here it's relative to the fixation point, and it's also oriented relative to the viewer, I'm going to talk about the two sides for now, but so I'll come back to this figure.

A

Soon um we're doing all this talking about optic flow, I figured it was useful for me to show what the first video I saw that just made vivid to me how good our optic flow system is. This is from mike castlemo. He did this in our presentation where just simulated a 3d environment with dots randomly distributed around it uh and you can see his laser pointer shining around. Oh I'm sorry um here I should redo this with uh with zoom set to update.

B

They work pretty well.

A

Hopefully, it's good so yeah the the fact that we immediately can understand this room and understand all these points on this wall. As things move around um to me, this just suddenly made it clear that optic flow is something we're very good at and it's it's probably really important how visual processing works. um My only point here is just to like let the audience in on this, like we keep saying the words optic love, but if you've never seen a video like this, you might not understand the importance of it. uh So just.

B

I mean this is a. This is a visual example. When I show the picture of like the video game- and I say you can say like you- can determine the movement of the player up down left right backward forward. All those things are pretty obvious, and this is just a video of that. I guess.

A

But also just the fact that just a bunch of randomly distributed dots are enough to give you the perception. Yeah.

B

But um yeah, but that's but in the end that's all yeah. I guess that was sort of impossible.

C

There's no local, I think the point is the information is purely in the motion. There's nothing yeah.

B

I got that that was my step.

C

Yeah there are more dramatic examples where, if you stop the movement, you literally cannot tell the shape at all yeah.

B

Here, if you stop the movement.

C

You can still see the shape.

B

But other places where.

A

When you stop, it's just.

C

Looks random and as soon as it.

B

C

Suddenly the 3d shape pops up yeah.

B

I guess, even though I showed it with a video again, my assumption was always it's pure optic flow that you know this, that you don't need to have you don't need to have features to do this right, which is just showing that, but that that was been always my assumption, that um that it, the optic flow, is efficient, and this still illustrates that.

C

uh Up to a point, I think.

B

Yeah and then, if you have.

C

Features of course, the visual system will use those cues too, where it's sort of amazingly. Well, I don't know.

B

About that, because because I think the way it works, if we think about the magno and the proper cellular layers and the argument that the the magnetocellular layers are doing, the optic flow, they don't detect features that you've got these two visual systems right and so the one that the one that preventing is presenting optic flow is not is not feature based and the features the the proper cellular layer. They don't respond during optic flow. They they only respond when it stops.

B

So it to me that was like evidence that there's like two separate worlds: there that we're really not looking at features that, when you're moving, it's really just optic flow bits that are that are paramount, because, because apartment cellular array doesn't respond, when things are flowing, it has to stop moving.

C

The magno pathway, detects edges and stuff.

B

They show that it's, it responds best actually for random bit patterns, the edges. They they are. The arguments the most recent or the arguments were that actually people thought it was going to be edges and they do respond to edges, but they actually show that it's, it's not the edges that are important. It's the it's, in fact, it responds better to random bit patterns and um and they they thought it was edges because they started by you know like hubble and bezel.

B

They were detecting, feature edges and that's what they assumed, what it was, but then they showed later that it's actually it works best when you don't have any features that the edges is a the edges are just uh our interpretation of of what it should look like. I think that's what the data shows.

B

E

A

Point I mean the point of this: wasn't that it was an aside to give people a mental image.

B

I thought it would just.

A

Be helpful to some yeah uh so bringing back the the figure um I was making the distinction uh that the I thought this was useful terminology that ballard uses um he describes. These fixation relative coordinates. These world-centered coordinates as they're world centered, but they're, not world oriented, they're world-centered, but they're viewer oriented so.

B

You're saying it's: it's it's uh anchored to the top of the tea kettle. Teapot yeah, but it's it. Its angle is, is relative to my case. Is that what you're saying yeah.

A

Because often we're struggling for a way to describe similar things not really to the picture here. But when we talk about object, vector cells, object, vector cells are world oriented, um it's a it's. You could describe them as either viewer, centered or object-centered, but.

C

A

Point is this notion of how is something centered and how is it oriented I felt like that was that was a useful distinction or useful piece of terminology. I might try to use that more saying how it's oriented, how it's centered um so okay, terminology aside this is this- is what ballard's proposing the visual system is using that uh I've. I have another slide, that's going to continue with the same figure. I'll just make sure I made this.

B

Does he say that that the c and c prime don't exist or that they're both b and c.

A

C

A

um Let's see the cmc prime I'm not yet ready to talk about it he's it depends on what you're reading I think they exist, um but he's more focused on the on.

B

Yeah, because if he said that c and t prompt didn't exist, I would really question that.

A

No, I I he doesn't make that statement.

B

A

He doesn't make that statement, but I'm not really prepared to describe his viewpoint on cnc prime today. Well.

B

This to me, it seems that you have to have some some type of object-oriented references reference frame to make predictions from novel situations. It just it seems like a absolute requirement. We don't have to say what that reference frame looks like, but it has to be some sort of reference frame anchor to the objects to make predictions about that object from novel uh poses so, but all right he's not making that claim. So, let's keep going.

A

Right, but let's make sure I say these so so the thing um it truly is a like a reference frame based on this fixation point where it really is capturing depth information, it really is capturing 3d locations and the way the way that that's all calculated uh here, I'm still going to continue emphasizing the flow here, the the way that it calculates 3d location relative to the fixation point. um My next slide will give a mechanism in these systems in an inanimate vision system when you fixate on a point for each patch of the retina.

A

The cortex detects depth information for that for that patch and that depth is relative to the fixation point. I'm restating what I've said before. Just in a couple more details, you you, you focus on a point that one point I mean you're fixated on it everything else you're, seeing now relative to that point, where uh how how.

B

Close, it is to you relative to that.

A

Point or sorry I should have said uh you're inferring locations relative to that point and the the.

B

Referring allocations, I mean one is the for my replication of my head or my eye, but you're saying even like the tea kettle or the teapot points are going to be relative to that fixation. Point.

A

At the very least, when you're fixated on, like the top of the tea kettle, um all of the points on the tea kettle you're gonna, be see whether they're in front of that point behind that point, uh but even going further uh seeing the coffee cup at near periphery, uh that will also.

B

Okay, so it's not like I mean you can think of this, like a where pathway, you're, trying to figure out where something is relative to your body, but we're not trying to figure out where the features of the coffee cup were relative to each other. Necessarily, this is all like distance and direction from my gaze. Maybe like is it in front of? Is this in front of this, or back of that I mean to say that the coffee cup is closer. It's just relative to where my head is at this point in time.

B

um It's not closer in any absolute sense right.

A

Sure, in the sense like in the sense that it's not good for learning this arrangement, yeah.

B

A

In an alice centric way, yes, this is still all viewer oriented um yeah. This is still all viewer oriented. I.

B

A

The motivations here are not yet object. Learning the motivations here are bottom up: visual processing and flow information and figuring out how you infer depth in a fixating visual system. So if I were giving a presentation on object, modeling uh I'd I'd have to fill in some more blanks. Here I mean.

B

In some sense, the model that you've done- I presented again recently with the the graph. You know we say the node has a. uh We have an observer looking at the node in the graph or looking at an object as some pose.

B

What we have to calculate is the the distance and direction from the observer to that object and, and the direction is a tricky thing for visual systems to detect, um and so there's got to be multiple there's multiple ways it does it and you're saying this is one of the ways the vision system could detect the distance to something um it's a component of the modeling system that we we've been working on.

B

It's a required component of some sort.

A

So the diagram he gives for how this works uh so consider the scenario of moving your head while fixating on a point.

A

Everything when you move, if you fixate on a point, everything that is in front of that point, moves in one direction and everything. That's behind that point moves another direction. It moves the opposite direction and, uh and the apparent velocity, the apparent velocity of those points is proportional to the distance. From the from the fixation point, uh and- and under these uh I, this is a direct quote from the paper under.

E

A

Conditions, it's easy to compute the scaled depth and scale depth here is um suppose you don't have enough information to know how far away that fixation point is you can still infer a useful quantity. You can still infer uh this quantity of the depth divided by the fixation, though even.

B

If it's ambiguous.

A

How far away the thing.

B

What's the difference between depth and fixation depth, I'm missing.

A

um The the depth of that patch of retina or the depth like you're fixating on I don't know your computer and in your periphery, you see a coffee cup. The depth is the depth of the coffee cup. The fixation depth is the depth of your computer they're, just you're fixated on a depth and meanwhile you're also computing the depth of a lot of other things. I.

B

I just didn't: I just didn't know what the definition of depth versus fixation depth I mean is, is like depth to coffee cup depth and fixation depth. The point I'm fixating it is that yes, so so depth varies depending on the different object. So all you basically saying you. If you don't know how your head is moving, you can at least calculate the relative distances of two objects, something like.

A

That, okay, yes, exactly that and that's exactly the way to say it. If, even if you don't know how the head is moving, even if you're not receiving proprioceptive like knock.

B

A

Appropriations up to invoke you can still compute this. The scaled.

B

C

That so that kind of goes with the.

A

Video game example too. This notion of how far away something is um that might not always be an answerable question, but this quote-unquote scale depth is always an answerable question.

B

A

uh So this to me, I mean this. This is the this is the the idea, some part, one summary envision: systems with gaze control uh for each retinal patch. The brain should detect a world-centric, pure-oriented location relative to the fixation point.

B

Why did we even need to invoke our reference frames in this case? All you really need to invoke is relative depth. That's it it's just say. If I want to know the relative depth of things, I can look at the flow bits.

B

Why do I even need to think about reference frames? It seems like it's so far you haven't used it.

B

um You know it's b. Why do I have to have a b? All I have to know is that I'm just trying to figure out the distance to b. I don't need a reference for him to be.

A

Now, I'm I I feel like you're coming at this with a point of view on what a reference frame is.

B

I I guess I'm saying I'm trying I'm trying maybe incorrectly bucketize this whole thing and say: oh, this is a cool technique for how we perceive depth between objects or how far something is, and I get it, and but to do that. All I need to do is fixate on a point move my body and I will determine relative depths of all the things in the scene yeah, um but I didn't need a reference frame. I didn't need like a there was a picture showing the reference frame.

B

You know, b on the top of the tea kettle for the teapot and, and that really was necessary. I just all I have to do is say fixated point move took it optic flow and I got my relative depths if you want to call that reference frame the depth reference frame, but it's fine, but it's just one dimensional, it's not like a you know. You showed two axes there, but if it just had one axis, all I'm doing is trying to figure out where something is along.

B

My gaze of sight and that's my that's the reference frame and so okay. I get that, but I it was implied that there was like a 3d reference frame anchored at the top of the the coffee pot. You know what I'm saying, uh but I don't really. We don't really invoking that we're just invoking the depth. One dimension.

A

So the um I I guess when you have a retina topic map here, I'm using map in like the sense of that neuroscientists use it retinotopic map the the full 3d location is captured by which patch of retina are you and what is the depth.

A

uh So in that sense it is a reference frame. It is.

B

uh Well, each patch of retina is uh it's just only described as a mechanism to saying in my patch what is the depth in my patch now, if I would assume that I all the patches know their relative arrangement, you know how they're splitting out in space fine um but but the patch on its own to say assuming I'm a patch of v1, I'm getting part of the retinal input, and I don't know about anyone else. All I can do is see what I can see.

B

um Then I would what I've determined here is what I've determined is. Well, I'm looking at something- and this is how far away it is.

A

B

Fair, but that.

A

That's, I guess, in the point of view of ballard and others, that's kind of how the visual cortex does these reference frames yeah.

B

But it's almost assuming you know yeah, but.

A

That's that's what it takes well.

B

But there's this constant idea- and I don't know if ballot has it that there's this sort of you know image on the back of the retina and and each part of the image is looking. You know it's a range looking at a different array of the world and therefore, if I know the depth of each part, I built the map, but the cortical column hypothesis says no, the other columns. I actually don't know what the other columns are.

B

Looking at, it's more like fingers, you know my fingers each have some sensation, but they don't always know what the other fingers are where they are right. It's you can't say that they're, you know in some sense each column has to figure out on its own, um and you know it does that through movement as opposed to knowing what its neighbors are looking at.

B

So to me, one of the freeing ideas of thousand brains theory is that you don't have the columns, don't need to know what the other columns where they're looking at the world they don't have to know that they just can look at their own input and say: okay, give them my own put.

B

So we would substitute like this method would say. Oh how far away is this thing for me, but that's all I know and and then, as I move and look at other things, and I know how far I've moved when I looked at something else, I can build up this model, but I don't know what my neighbors are looking at right. I have to work. If I'm looking through a straw, it has to still work that way.

B

It's just a it's an important but uh distinction. I guess.

A

B

I yeah, I guess the the.

A

Main point I wanted to make in this part one that I'm about to move on from is: um is that ballard talks, a lot about reference frames and and world-centered coordinates and is for totally different reasons than us, but they're ones. That, I think, are it's a it's an area that I hadn't considered moving around while fixating and and computing depth is a really interesting scenario that actually might be important to consider for understanding.

B

Yeah and I'm just I'm, I'm making the point, and maybe I'll have to leave it this, I'm making the argument that it's wrong to think of it as a reference frame. It's it's just a method of determining depth and- um and you don't need to invoke reference frames at all to do this. You know not the kind of reference names we talk about which are like you know, multi-dimensional.

B

It's just it's a method to show me depth.

A

Okay, with that I'll move on to the.

C

Second, yeah: okay: does he consider stereo vision.

D

At all, because.

E

That's an alternate way of.

C

Getting depth so is this: something is the idea that this would work in concert with a stereo vision system to get more accurate depth. Yes,.

A

Basically, I I chose not to uh emphasize the stereo too heavily just because we know that the vision system works without stereo vision. We.

B

Also, but we also know it works without moving your head, so so I mean.

A

But that but like you can be born with one eye and be fine, you can't have a developed vision system without moving your head.

B

No, but moving your head in two different ways like obviously you have to move around, um but the question is: are you reliant on the flow bits as you're fixating? um Are there other ways to get depth? You know, and we can argue whether you have to learn how to do this or not, but in the end a lot of vision can occur very important business without this technique. So it it this. It can't be completely reliable.

B

um It's just like it's one of n ways determining depth, and maybe maybe it's the most important one.

D

There's a couple decades ago there was a 3d system that worked only on this. You were looking at the screen, but what they would do is they would take the pixels that were in the foreground and jiggle them at a at a larger displacement than the ones in the background, so they basically encoded depth with the amount of induced motion on those pixels, and that was enough to produce a perception of depth. So you were in cicada you weren't fixating, but the motion in there was sufficient to elicit a uh a sensation of depth.

D

Well, that's the same.

B

Basic principle, though, right.

D

Right but I mean it stands alone on, on its own, I mean even.

B

Though you're fixated but that's an artificial uh system that I mean that would.

D

Have still perceived it as what I'm saying, yeah.

B

I know so I mean very essential yeah, so I think you're arguing that that this thing that ballot and marcus just described explains why that works.

C

But but that's not a natural.

B

Way to happen, that's not going to happen in life, it's just not the way. Well,.

D

I'm just saying that whatever system we have in the visual system is excited by that alone, even though you're not coherently moving around making that motion happen. Yeah.

B

Yeah the fact that it's doing it all yeah, but it's the same. Basically, you know there's there's an argument made, and this might be true. I don't know you know we have microsoft. So there's this idea that even when you're fixated as hard as you try not to move your eyes, your eyes are moving and um very very small amounts. They kind of like the the fixation point drifts and then it snaps back and then it drifts and smacks a very small amount, but enough.

B

The microphone circads are enough to cause uh the changes of activity on the ganglion cells coming out of the rib, so microsoft codes have their physiological effect that the brain has to do with and one argument for them. I think was this argument that um so even when we think we're fixating uh we're actually moving somewhat and maybe um but you know that that wouldn't work in this card, because for this, for this effect you just described marcus.

B

um You won't get this depth effect by just moving your eyes. You have to move your body if you just move your eyes all the pixel relatives, each other stay the same right. So um so I'll take that back. um I was thinking that I I thought someone once made an argument that microsoft helped. You perceive death, but I don't see how that is anyway.

A

I don't have the figure handy, but one of the points balor does make, as um as uh as we know, and often lament uh a lot of the recordings and neurons were happened and anesthetized animals and the receptive fields of these cells might be better understood and there's there. He points to some evidence, for this might be better understood in the scenario of moving while fixating and understanding how the cells respond in those uh in those contexts.

A

um There are reasons to believe that that's actually that's actually like really central to how these um these visual cortex cells respond, like just even just like center, surround like what happens when um you have movement in one direction in the on region and one the other direction, the off region. I'm sorry.

D

A

Should have actually prepared the figure for this uh larger point is um uh examining these anesthetized animals. You totally missed this scenario of.

B

Yeah moving around.

A

While fixating.

B

One more thing we could throw in a bigger point, one more thing: we throw in the mix yeah. Another data point is: what can you do when you're looking at a printed image and, of course, when you're looking at a printed image, none of these, these stereo effects can occur at all right um and we're pretty damn good at it. You know it's like um uh you know, so, that's obviously you're picking up environmental clues, there's no flow information whatsoever to be.

A

B

A

Interesting that you're so focused on this, it seems like that's the result of a really well trained division system. Well,.

B

Maybe, but in the end, what the point is it might be, but still it points that that the mechanisms will flow are not essential during the inference phase and not essential during learning. It might have been essential to set up the system that way, but it's it's an it's an important point to make that to know that distinction that I can still learn a very complex um scene or model of something without any flaw.

B

Information whatsoever- and yes, I may have had to have prior training to do it, but it just tells me that the mechanism isn't, you know absolutely essential for these processes.

A

Would you be open to the idea like to to maybe brand that make this a thousand brains thing like some of the brains are inferring depth, information from flow uh from flow, while others have learned these other advanced strategies that involve cues within the image, so you both are going on in parts and parts in some of the columns.

A

The flow flow is very much driving it and others. It's video.

B

I totally, but in fact you know you recall, I've made the argument. That flow is how we learn uh what movements are possible in the world right that and that, and I've made the argument that many columns are representing some of some aspects. Some layers of cells and minicoms are representing uh different flow vectors, which are different movement vectors. So to me, the integration of flow into learning about the world is absolutely essential.

B

You can't even learn about behaviors, I think until you've observed how your sensory flow goes, that they define how you're able to move flow information is the definition of what behaviors you are able to achieve, and so I believe, they're absolutely essential. I'm just trying the only thing I'm trying to make a distinction is that when we have to explain how it is, we can also do these things when we're not. We don't have low information after the fact.

B

That's the only point, I'm making it's not the diminished importance of it. It's just to be aware that there's that distinction.

C

Yeah and even after we've learned about the world, we can, we can learn about new environments with just a static image. We don't have to yeah.

B

C

Have to learn, you know, even in a new environment, we don't have to move around to learn about it.

B

Yeah, so that's it just! It just puts another constraint on our learning systems to say, okay. Well, I have to be able to explain.

C

That we can do that with a 2d image where there's no, you know, there's.

B

C

Depth movements, nothing yeah.

A

Everyone agrees on that point yeah, so.

B

A

C

Diminished model in his model for inference, can you do it? Is all this required for inference as well.

A

In this particular model that was aimed at flow based depth detection I mean no, it doesn't have that built in uh that, so but that's just because that's not what it was designed for. It's just a robotic system that tries to model some of the visual cortex's uh representations.

B

In some sense, ivan argument flow is even more essential than he's arguing here. I've been saying flow is not just determining depth flow is determining all possible movements in the world, and uh it's it is. It defines in some sense a movement-oriented reference frame which I'm still working on an idea. So um it's it's. It's a I'm not arguing against the importance of law. I think it's even more important than this.

A

Okay, I think now maybe I'll move on to the second.

B

C

Sure that has nothing.

A

To do with flow, uh okay, okay, so stepping back um animate vision is a system that generates its own movement, uh so this um yeah I'll describe what what I've drawn here on the left is a figure from our paper who we often call the columns plus, where we solved the problem or attacked the problem of of a system that receives as input a sensory motor sequence and from that recognizes objects.

A

But I mean in our minds the larger system on the right. The frameworks paper.

A

This is part of a system that you know is generating movement, and um would we have landed on a different solution on the left if we'd embraced movement generation more or are there? Are there opportunities that we've that we've missed here?

A

uh This is a question that um that I think is useful to ask and that it's thoughts that are provoked by balor's ideas. Some of them are some of them are things he said explicitly. Some are just following the pattern of thought and uh so yeah two different. This is just a restatement two different sensory motor jumping off points.

A

Do they lead to different object, modeling solutions and one the motor command is being received by the system and the other is being generated, as jeff mentioned, the superior colliculus generates motor commands and the visual cortex will receive those. In my mind, this black box contains the entire brain. Essentially, if you just focus on the problem of what is the brain, doing it's generating uh eye movements and in order to obtain information really quickly. uh If you try to solve that problem, what.

C

A

Of object, modeling system: do you need fundamentally, so there are multiple ways that using the control of movement could make the model's job easier. The the job of of what you need to learn. If you can control movement, how you can kind of optimize that make the learning problem easier.

A

Simplification, one uh is with gaze control with the ability to move the eye. This is something I said a little bit um where my terminology kind of clashed with the subutai a little bit, but where you know you don't need to support cicada into arbitrary parts of of an object. You don't have to be able to sacada any single point on the coffee cup and predict what you're gonna sense.

A

um It is sufficient to saccode to uh to particular points on it to have like an idiomatic way of like now, I'm gonna move to the upper right corner of the coffee cup. You have some control over where you staccato you.

C

A

To your strengths, you succumb to the parts where you're going to be able to make predictions, uh and that is that is an important thing. You don't necessarily have to be able to scott in any arbitrary way and make a prediction of what you're doing, but.

B

There's an important distinction again similar to what we just argued earlier is like. Can the system make a prediction to any arbitrary point, and is that or it or not, and is it just much more efficient not to do the arbitrary points? um So one is an efficiency argument, the other is no, you don't have to be able to make a prediction. Any arbitrary point.

B

I think the I think the former is easy to argue for. The second is um I mean I think in my mind: it's always you always have to be able to say yeah. I don't normally look over here, but if I do look over here, I should be able to predict. What's going to be there um as opposed to I just it's not very efficient to do that.

B

Yeah or you know what I'm saying so when you say no need to support that, you could go either way right either it is, you can still do it, it's just not very useful or no, it wouldn't work at all. Therefore, you know we're not going to rely on it. It wouldn't work at all.

B

I think that's harder to argue for.

A

Right so in my mental model, the the problem of, or rather the um I don't know how to say that right I'll start that sentence over if you're the job of recognizing objects um like distinguishing this coffee cup from a thousand other things that you might be sensing um that recognition process might require. Like specific cicada movements, you might not be able to just do it from a randomly generated set of cicadas onto the cup.

A

uh Once you recognize the object once you recognize that this is this coffee cup, I think the prediction becomes a much more um tractable thing where you can stick out at any point.

B

So I mean when you're talking about inference. You know the prediction like yeah. Imagine I'm touching, I'm touching a cylinder. Am I moving my finger around the cylinder? Well, I don't know if it's a coffee cup or a tube or you know a coffee pot or you know you know a drinking glass or something because they all have the cylinder.

B

uh And so you don't get any more information from just running your finger around the outside of the cylinder. You have to go up and feel the handle or feel the lip or you know some.

B

It's not a great example, but my point is obviously there's some components of an object that are much more valuable for inference and we we show this in in the comsplus paper that you know the the more unique a feature is the more useful it is for determining um inferring the object right so, but so I I don't know if that's a fundamental point other than just say yeah, some some sensations are not going to be that useful and others will be much more useful.

B

But I don't know if it's it's a broader statement than that. um You know it's not like it's not useful at all. I just it just seems to be yeah some. Some some features are going to be more unique than other features and therefore they'll tell you something.

A

Yeah uh the second um well, this is a this. Is the less important point um I guess to answer your question. I was just making sure I'd showed all the bullet points for this.

D

A

Level, so um I guess one of if you, um if you're, not careful on how you frame this problem of of learning, to take columns plus, for example, we were.

C

A

These objects by learning features at various points.

A

C

A

Now, I'm just repeating things we said toward the beginning.

C

A

Presentation: it's just that you can use a smaller set of points because you have this ability to change what you're fixing.

B

Yeah, I think I think we captured that in the columns plus paper in some sense right.

A

Yeah we did it, you know. The funny thing is at the time I didn't understand. I thought that we, it was actually a flaw in the model, but it wasn't a flaw in the model.

A

It's actually a reasonable assumption. I.

B

Mean I don't think, we've actually done this. I think what you're saying- and I think it's a good point is unless let's say we did columns plus again and we we built the models of the objects using. You know the newer, our newer thinking right. Let's take we adopt the the vector cells and the graph idea and, and then you're saying, okay. Well, I've sensed something um at some point and now I have like this three objects.

B

This could be, let's say, I'm down to three things and then, which way you move you could from that node on the on the graph you could say: well, um I'm not going to move arbitrarily, I'm going to move in the direction which takes me to um you know a feature that I've I've learned you know uh and uh which is like just still keep going around the cylinder. The coffee got.

B

Let's go up to the lip, but let's go out to the handle and see if there's a handle there, so you say well if this is a coffee cup. If I move this direction according to the graph, I should see a handle. So, let's check that right away yeah, so I think that's what you're saying is. We can be smart by following the graph edges um in a in a way that says yeah.

B

Well, that's our knowledge about the object, therefore, we'll test it by by following graph edges and not just going off into space yeah.

A

So so you, basically you basically called the seconds uh simplification number two. Now that I brought that bullet point in um the the the order in which you sacad over the points so so you'll you'll see you'll see. The theme here is the first one is the set of things that you succub to the set of points on the object can be like finite and not any particular point.

A

The second is: what order do you look at the? Do you look at them, uh so the the point I want to make here is like uh in columns plus we solve the problem of being able to cicada over the points on an object in any order.

A

It's not specific to the sensory motor sequence, um but if you have control over at least strictly speaking as it doesn't necessarily need to be solved strictly speaking, if you have control over your staccato order,.

C

You don't necessarily have to.

A

Be able to handle any order of the cicadas and the reason this is well. One reason this is a important insight is with this graph based model of objects. If we, if, if we really do, learn graph based models of objects, you can't second in any arbitrary order uh unless the, unless the graph is densely connected. Unless every points, unless you have relation info for every pair of of nodes, uh you have a.

A

If you have a sparse graph, then you can't second in any order, but that is not a um that is not a fatal flaw if you control the cicada so like.

C

You have to be able to. You have to be able to support a lot of different card orders. It's not always the exact same order. Each time.

A

I would say the I would say.

A

The it can support a lot of saccade orders, but all of the first order transitions need to be learned every pair moving from the handle to the lip of the cup, um I mean that's just by definition, a graph. If you have this, if your model is based on nodes and relations, then the only thing you're going to make be able to make predictions on is when you're moving along the edges, which jeff was just saying.

C

Are you thinking of everything from a single cortical column, point of view, uh because with multiple cortical columns I mean again you can learn things without any saccades, so I'm a little I'm a little confused yeah.

A

Okay, uh I guess I was focused on a single cortical column. um If you, if we want to have a conversation about the multiple columns, we can.

C

I think this is where I'm confused about things like no need to handle translations of the features and things like that. I'm a little unsure about the setup here.

B

I yeah, I think this multiple columns question is really really hard in some sense. It's what I'm working on right now and I think we're gonna. It's we don't there's this idea of attending two different components. One at a time using multiple columns is is, is something we have really figured out exactly how it works. That's my I'm going to contend um so I think it's more fruitful think about just think about it. Abstractly at the moment, if you will um that's, why I'm doing it, at least I I think what you're saying.

B

Let me just try to make a a an analogy that we can all think about here. I've used this one before, let's say I'm, I'm looking at a room and there's all these different. It's a big room. I know that's in furniture. I and I want to learn the graph right. I don't learn the graph between every object in the room.

B

That's a two highly connected thing: there's dozens of objects in this room, there's no point in learning the distance direction between the you know, the the the vas on the shelf and the chair in the opposite corner of the room. um So what we tend to do is we tend to learn the local connections.

B

It's a small world type of thing where we go oh from the bars to the shelf, maybe the shelf to the bookcase and maybe the bookcase to the to the wall, and you know this kind of stuff I could. I could gauge from go immediately from the vas to the chair in the opposite end of the room, but I it wouldn't be very useful because those positions to each other might change all the time and it would be very difficult to learn it and there's just too many things like that.

B

So um I think what we typically do on a real object. Is we move consecutively through local features that are near each other and those are the edges of the graph we learned and when we're trying to infer that's what we do too. We don't we're not going to jump all over the place right. You know we're going to sort of follow the sort of locally adjacent features or uh uh nodes on the graph.

B

um So again, so we don't if we don't have to support arbitrary chicago, of course, because it's it's impractical to do that. I mean it's. It's it's not like we're going to get confused if I succumb from one corner of the room to the other. It's just not going to be this helpful as if I sort of see things local to other things. First, does that make sense at all.

A

Yeah I mean that fits with what I'm saying.

B

A

Yeah and I'm just making the simple, maybe maybe the statement I'm making is almost like too simple? It's just um the our brains, don't have to learn models that uh that enable just random fixation like you, you fixate to some random direction in this room, and then it then a different random direction in this room and then a different random direction.

A

For all. We know our brains models might be totally incapable of recognizing the room.

B

I think I think that'd be a very difficult way to learn, um but I think our brains would be good at inferring. That way and the.

C

Reason yeah, I I think we could infer that way.

A

And here's why and that's let's say in the scenario of looking through a straw like where you're yeah.

B

You're looking.

A

At one random thing and yeah.

B

Here: here's here's: how.

A

Do you have evidence for this.

B

Yeah, well, here's how it can be helpful again. I look at something I don't just I I make a hypothesis about what the room is or what environment I'm like what object. I am. I have a hypothesis by attending to some feature now, if I zoom some far away and I get another input and I don't- I haven't learned the edge of the graph, so I I necessarily can't predict that future, but I get I get.

B

I see the other feature and I get a hypothesis about what rumor meant by not based on that other feature, and um I can then decide based on my two hypotheses. That is this the same room or is it you know? Is there still have? I eliminated the rooms or not? It's not because I know exactly where the vibes on the shelf is relative to the chair, but it's like that. Violence only exists in certain rooms and the chair only exists in certain rooms, and so um I can do this by process.

B

You know voting and union elimination, um so it's still useful for inferring, even though I haven't learned that particular edge, I don't need to learn the edge I just. I can still eliminate possibilities just because I've observed features that are not ubiquitous in the world. uh They don't exist in certain places. um So that's that's my only point there.

B

I may not be able to predict exactly what I'm going to see in the far corner, um because maybe I maybe I see the thing on the shelf and I said well, I could be in one of four different rooms, so I can't really predict what's going to be in the corner, but I zoom over there and then you know it's our standard voting thing. I see something in the corner. Well that only exists in one of those rooms. So therefore I know what room I'm in.

A

So I yeah that that that fits with what I'm saying, I guess the the statement I'm making where I I'm coming at this from a position of humility where there's a there's, a there's, a point that I don't know, there's a there's a thing I that I wish. I I don't know how you would how you would test this, but if I could take control of your vision system and your your cicadas, your everything and and I could control um I'm going to put a familiar object in front of you that familiar object.

A

That requires multiple cicadas to to recognize and I'm just going to control you your vision system, I'm going to point it at this point and then a different point on the object and then a different point on the object. Will you recognize it well.

B

C

Yeah, you would have to like tell me: okay, now, look up into the left, yeah and now look up into the bottom, and so.

A

On yeah, I would imagine I'd be able to recognize it. It may not be.

C

If it's random, it may not be very efficient because a lot of times you know I may not get to you know if I'm looking at one blank patch of the wall to another blank patch of the wall, it's not going to help me percentage stuff.

A

Where's your confidence in this coming from, though.

D

A

Our vision system does generate its motor commands so um yeah the question: this is a hypothesis I wish. I knew this experiment. Is this just your hunch, or do you have a reason for being confident of it?.

C

What I think well there's a difference from I mean this. This two separate things. Yes, my vision system is generating my sakas and yes, I tend to. We tend to look at specific types of features and there's you know different patterns there. It's not totally random, but that's very different from saying. Oh, if it was more random you'd not be able to recognize anything at all.

C

um I I I'm not sure why we need to go to that extent and I'm pretty sure we would be able to as long as again as long as I'm the one generating my commands. uh As long as I know what the motor commands are, I should be able to do path, integration and recognize things.

B

Yeah, I think I think this we can go into the evidence, but I think we also have to bear in mind that you know it varies. If I'm looking through a small straw and I move long distances well path, integration's not gonna work, it's just it's gonna, be too error-prone, you know it's just you just you! You're gonna very quickly lose like how far did I go. You know, while you're, if all you're getting is flow bits you know through your straw I mean, and so I think, there's a you know. There's.

C

No, you have more than that, because you know what I command. You used to.

B

All right, well, anyway, regardless I think path, integration is very noisy. So I just want to point out that you know if marcus may be thinking like okay you're, looking through a straw and you're, seeing one little feature in the corner of the room and you zoom over to the other corner room. It's not going to work. I'm saying I don't know if that's going to work because it um because it's you know, there's a there's a limits to what you a single column can do and how far can path integrate and so on.

B

um I just want to point that out, uh but I think in general I I'm surprised you're questioning it marcus. It seems uh it seems to me and I could be wrong, but it seems to me that we can do this. I give the example of how just um I you know.

B

I I give the example where I showed you a feature in the room and then another feature, um and I don't even have to know even a bag of features can solve that problem. Sometimes I don't have to know any movement at all. There's only one room in my house that has that box in this chair you know um so there are different levels of which these inputs can be useful.

B

A

uh I mean yeah, I guess my sense is, but my real answer is, I don't know it to me. There's this logical possibility that the like the brain is all about being economical and and doing the minimal thing that it needs to do, and this economical trick for recognizing objects um involves.

B

Yeah, I don't think there's any disagreement with that. The question is, the only question is: um uh is it uh is it? Is it the limit? Like? Am I unable to make other predictions or does it limit what the model can do or is it? Is it a fundamental limit, or is it just an efficiency argument and I've always felt it's an efficiency argument um for the most part I mean, obviously, if I'm sensing something and I don't sense- the relevant, important components of it, I will not learn a good model of that object.

B

I may not learn it at all right if I'm looking at someone's face- and all I see is their hair. Well, I'm not going to be very, and I ever I learned everybody's a model of everyone just by their hair. I'm not going to be as good as recognizing faces.

B

You know, so you have to there's some intelligence involved there, um but, but fundamentally the algorithm, I think, is the same. I don't I I just it's an efficiency uh argument. In my mind: it's not um it. Doesn't I made the same thing about the the depth perception. I think that's an efficiency argument, not a fundamental argument.

B

No, I I'm making that argument because that's a theorist. We might like to believe that, but maybe you know maybe it's wrong, but that's that's the argument.

D

Marcus have a question about your your graph model. Are you assuming that you always start the graph at a single start, node and then have to reverse it in that particular order? No.

A

uh It's just like the first order. Transitions need to.

B

Kevin, you don't have to start in any particular place and you don't have to traverse in any order.

D

Okay, because I I was, I was hearing that the learned order was something that was being.

B

No, it's just all you're doing is learning the transition between two, like the the relative positions and orientations of two things, and um and and the graph can be stitched together anyway, and starting any place going anywhere.

D

Okay, so if you hypothesize that you start with the most prominent feature in your visual field, start from there, you can transition and fill out the rest of the picture by looking for possibilities to complete.

B

Yeah, you don't know you don't even have to start the most prominent. You can start anywhere.

D

B

D

Could yeah there's.

B

No harmonization to.

D

Start the most prominent, no.

B

Wait but who just say what's most prominent, I mean in some sense, if you know in advance, you could say. Yes, these are the things that are important and I'm only going to focus on those things, but um but that's uh I think we don't want to make that assumption. We just want to say look. Obviously you don't want to gaze randomly around the world either. We don't do that. It's obvious right that we we fixate on not always, but sometimes we fixate randomly.

B

It looks like, but often we come back to the same features over and over again, um like the fight the face, you typically go from eye to eye to nose and out the eyes item those and that kind of thing, okay, but it should work. It should work in any order. In fact, one of the one of the the learning rules of neural neural circuits is if, if, if a particular edge graph doesn't appear over and over again, it will be forgotten right.

B

So if I learn the particular location of a of a curl in your hair with another curl on the other side of your head. Well, that's not very useful and it's going to change, and so, if I fix fade between those two and for a moment at that moment, I'll say there's the edge between those two.

B

But it's it's going to be forgotten because it's not going to repeat it's not useful, but the position of your two eyes is going to be it's going to be very static and that's going to be trained again and again.

D

Okay, so the discussion where you were talking about the order of saccades being somehow canonical that had nothing to do with the actual transitions that we're talking about here.

B

I don't think mark marcus, didn't say the order is canonical. He was saying: they're, not random, meaning like you're. Not it's not like. You can go in a specific order. It's like you're, not going to specific. Are you going to specific places or just random places.

D

B

D

Learned sequence of I'm sorry.

A

No just the first order transitions.

D

C

A

uh By the way this is this is wrong. There are 15 slides, but I'm I'm nearing the closing part of this. I figure I'll do the last two slides and then, if any communication happens soviet uh so to to um to give you a brief picture of the of where ballard goes with this, um he takes this in the previous slide.

A

I'm making the statement that when you control movement and you have the input um and you have sensory input and you control movement- you might land on different um different solutions on what the system should be doing. So now, just read this quote out loud. This is just I just took this from a random video. He says um before rod brooks it was the world of mar. Where vision happened and cognition happened, then action happened and the front end of vision. Was this principle of least commitment?

A

That's that's a term that mar uses uh the the principle of least commitment where you're just trying to build a generic description of what's out there um and then he says, like rod, rod brooks uh changed it all and said you should quantize things in terms of the entire task, with vision, hardwired right into action.

A

The broader idea, I'll read the bottom bullet point that I wrote this like.

A

If you the the broader principles, if you focus on making the system better able to perform tasks, uh you eliminate intermediate problems that you never needed to solve.

A

In the first place, I'm stating the where, where they come from I'd, say this is the ballard and rod book that they're that they would say that if you cut behavior generation out of it, you're gonna end up solving a different problem than you needed to solve, and I'm just presenting that as letting you know that, like here I'm talking about these views, I just wanted to let you know that.

B

A

This is their point of view and I think it's at least useful to keep in mind.

B

I, and sometimes I felt my mind this has been in the back of my mind, all along in the sense that all right we're not doing random movements. You know this is the there's reason. You know this, it's not random, um but we can just isolate that problem for the moment and not worry about it. That's the nut in our papers, we've written so far. That's what we've done right.

B

We've said: okay, we're not going to deal with that right now, but it's not also under the assumption that you're just throwing images on the retina and it doesn't really matter you know where they are or how you fix it or anything like that. I mean the whole idea of the graph. You know a model is that we have to tend to the features of an object, and so that means you're doing intelligent attention.

B

um It's just sort of assumed in the way we describe it. um So it was the hope that we can sort of separate out these two problems, but it's not like we're, not it's, and so I'm agreeing with them, but I'm also thinking. I don't think we have a problem with this. um In some sense, it's sort of the background assumption of how we think about these systems.

B

um It doesn't change fundamentally how I think about models. I guess, did it change how you felt about modeling markers.

A

uh um I'm still absorbing it uh not really.

A

The point I made previously about about maybe our object recognition actually does involve um succotting in particular patterns, even even if they're just first order. I'm I'm I'm open to the idea that that our vision system can't actually handle um arbitrary saccode patterns.

B

Okay, it's two things that one is: can it handle it and one is it do we do we, you know assume that that's you know. Do we assume that it's smart? I mean again. I think that we can assume that it's smart, but it should be able to handle random too um it shouldn't fail. It shouldn't be like if I gaze off into the middle of a wall of sudden I'm lost.

B

You know it's like okay, that that that fixation point, maybe wasn't very useful, can't do much with it, but it's not like the system's going to crash or something like that.

E

I think uh interesting research to look into this question. More could be on people who actually lost their peripheral field vision, because there are conditions where people literally just look through a straw at the world, because.

B

E

Only have like 15 degree vision. So as far as I know, they have considerable trouble um and I mean they have to make fixations basically into the black. So they don't really know where they will fix it to next. So, but I don't know how much it impacts their perception. They could be interesting to look into.

B

I mean that's a comment: what's the disease that causes that it's common in um in older people, is it like glaucoma or glaucoma.

E

Glaucoma and I think also retinitis pigmentosa, and there are several conditions I think that can cause us.

C

Yeah, I think I have a friend I think it might even be like retinal detachment or something that a lot of the retina stops working and he just can only see like a very small part of this.

B

Well, there's different, there's different things like my mother has naturally degeneration, so it actually parts of her phobia get damaged, and so she has to rely on peripheral vision, but I think people who have glaucoma it's just basically narrows your vision down to a very small section.

B

um I think the the biggest thing that they have there they can see, but they um it's they can't see things happening around them. I think it's like very dangerous to drive because you don't have you know you don't have a sense of things outside of your fixation point, but I think they generally have no problem at all, but I mean, I think, does anyone know I thought I thought you could read pretty well with glaucoma?

B

Is that not true? I don't know. I should check that.

E

Yeah, I was just thinking because they basically have to fixate into a black area for them, so that is kind of close to fixating to random.

B

But I don't that's not quite true because I mean like if I'm reading, even if I have a very narrow point of view, I'm not going to fix it randomly I'm fixate to the right. I'm reading english and I'll know how far I moved. It's like the brain doesn't know where you've gone. You've got this. um You at least have you know the the the you know, the the movement command that you're executing.

E

B

E

Motor and I was thinking more of like coming into a room and like more on broader skill for fixations, um trying to figure out where things are in the room. For example,.

B

E

In this case, it would be kind of isolated of having you have the motor command, but you didn't really purposefully put it somewhere.

B

So I'm reading I'm reading an article about glaucoma and reading right now and they say you would think that glaucoma wouldn't affect reading, but it does because glaucoma patients, uh often just their central vision, is also foggy. So it damages their central vision as well.

D

When you specifically look at tunnel vision, I think you get what the effect you want.

B

D

B

Say the result is slower reading, particularly with longer words all right anyway,.

A

So we just have a couple of minutes now my 15th of 14 slides uh the final one, um the things that I didn't make into this presentation. I could have dived deeper into the discussion of mar versus, brooks, which I mean this is like ballard frames it this way he says in that quote, it was like it was the world.

C

Of mar, then rod.

A

Brooks rescued us from it, that's kind of his viewpoint on this um that that you could go much further into that. Another thing.

C

That he says some.

A

Thought-Provoking things on, but doesn't dive deep into, is our models of objects, whether they're, you know I I don't even know the right terminology for this. I don't want to say he thinks that we don't have 3d models, but he has some, but he points out to some interesting ideas about features that can be recognized, that don't change when you see objects at different poses like the ellipse at the top of a coffee cup, but other other things as well, and that those might actually be important for object. Recognition.

A

uh There's, like I said, I'm listing the topics that I didn't dive into here. So that's about all. I can say there um and- and like I mentioned he, he makes statements about coffee cups in general and the degrees of freedom of coffee cups and how like, if a coffee cup is sitting on its side, there's really only a couple ways it might be sitting because the handle is going to prop it up anyway. He considers these different scenarios like that, which was just funny that someone else was talking about that.

B

You know, I think, there's a there's a really interesting point here, uh especially in the second one, and I've been really dealing with this a lot in the last three three or four days um is, you know when we think about 3d modeled, something it's sort of completely invariant to the pose of the viewer right it did.

B

A 3d model just makes no distinction which, while you're looking at it, it's it's just and you can just generate an image of it from any particular point of view, and we seem to have some capabilities like this, but it's not completely true. It seems like we do have biases towards the the poses and the viewpoints that we've seen and and I've made. The point part of this is because we can remember our specific viewpoints as part of our episodic memory. It's like I, you know.

B

Oh, I was standing here relative to the coffee cup when I, when I heard the phone ring, um but I think it's also. I think we have a balance between these two and um and I'll, throw out this crazy idea that I want to talk about later.

B

um I've been thinking about you know: do we really have a 3d model in the sense that we think like a 3d reference frame like 3d grid cells or um I've been kind of forcing myself into realizing that there's another way you could build the model to have 3d like uh structure but aren't in a classic. You know xyz type of 3d frame where you're. I don't know how to describe this, but we can warp space.

B

You can take a a two-dimensional grid cell model, which is like a a 2d reference frame and you can warp it. You can twist it or curve it, and- um and this I got to this by thinking about how the how the logo fits on the coffee cup. We take this two-dimensional model or logo, and yet now it's on the coffee cup and 3d it's wrapping around, and yet we have no problems making predictions about that.

B

um It's fundamentally breaks a three-dimensional model of the coffee cup, but if I had a two-dimensional model of the coffee cup and I'm wrapping it and I'm warping its space so that any local property in that space is still preserved, but the global properties are not, um then you can do this, so I'm using this as a teaser, because I think this question are there really three true 3d models is a is one that uh I used to believe. I'm not sure I believe anymore did any of that. Make sense to anyone.

B

Hopefully, stupid marcus anybody fit in I, okay, I guess not all right.

A

I nodded my head, I said like.

B

Okay, okay, I'm making an idea sounds.

C

Good, I guess my question would be like in the case of a coffee cup, I mean that's basically a 2d surface, we're talking about anyway. What about structures that are more, that really are 3d well.

B

I guess so the question I'm going down this crazy path of thinking about this way that there's there's two things when we take to modeling something one is: is the local relationship between features as as described in the graph, and the other is the space itself that those that that um structure is imposed upon and space can be dwarfed, and so it may turn out that the cylinder of a coffee cup is not a 3d structure.

B

uh It's a 2d structure that wraps around on itself in a space. So one way to think about this is grid cells themselves are like the fabric of space and the the links in the in the graph are just they're they're, just the local relationships between things and that the grid cells themselves may actually, as you move through space as you do path, integration- and I say I'm moving in a straight line- you may not be moving in a straight line. You may be moving in a straight line in the grid cell.

B

Reference frame like grid cells, say yes, but, as I move from point to point in a good cell space, the actual orientation of the grid cell module, the good cells themselves may be changing as you move, and so it you it's like it's like your path. Integration is moving through curved space, so you think locally, I'm moving straight, but but, as you go, it's constantly curving.

B

This means that grid cells would have to have like a they'd have to have a different set of parameters around they'd have to they'd have to as you they have to be able to curve. um I know this sounds crazy, but I kind of worked myself into this belief by thinking about this problem deeply, and it's like. Oh my god. This is a really hard problem. How do we take a shape and morph it into some other shape? And we seem to be able to do that quite a bit, so it may turn out.

B

We actually have no 3d bases. We actually only have 2d spaces that are being being manipulated and warped, um and so I wanted. I have questions about that. um This can mean that green cell modules really are 2d, but they can sometimes look like 3d, because they're they're they're warped.

B

They can- and I think, remember, marcus. You were originally talking about rats having 3d and said when they get to these obstacles. They kind of the good cells modules go up and over the hump. You remember that.

C

B

um And it reminded me of that- that's not where it came from. I said: well, that's kind of what you were describing and I think that actually might be going on.