Numenta Numenta Research Meetings, 29 Sep 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Discussion on Knowledge Transfer Among Cortical Columns - September 29, 2021

Description

Marcus Lewis frames the problem of knowledge transfer among cortical columns in the Thousand Brains Theory and explores potential solutions. He first explains how, at a high level, Numenta's model tackles this problem by having cortical columns communicate a description of an object horizontally through lateral connections.

Marcus then explains how this "horizontal description of an object" mechanism suggests a different mindset for the Thousand Brains Theory. Influenced by Douglas Hofstadter’s book “Gödel, Escher, Bach,” he states that a cortical column fundamentally needs to learn to represent descriptions of objects, so that, given a description of a novel object, it can make predictions and recognize that object no matter where the description comes from. Secondly, the cortical column can memorize those descriptions, but this object-memorizing functionality is secondary to being able to describe an object in the first place.

Jeff then dives into what information about the object is needed for columns to communicate and vote on what they’re sensing. He makes the case that columns communicate locally through lateral connections in the neocortex and work in parallel with each other. The team then asks questions and discusses.
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.

Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://numenta.com/news-digest/

Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter

Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta

Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/

Our Website:
https://numenta.com/

A

So here I am, this is a very informal presentation that we, I basically winged it this morning and decided um to write some stuff on the board and bring up a point. That's been in my head for quite some time. It's it's been like a sense. I had that we'd eventually go this direction, so I thought I'd finally bring it up.

A

It's come up in forms in the past, but I thought I'd um just give it to you fresh. So, um first of all, did you define the term knowledge transfer jeff used that term a few weeks ago, for the first time to describe a problem of I mean in the thousand brands theory you have this general.

B

Set of scenarios where.

A

One cortical column, you want it to use information that was learned or picked up from another cortical problem.

A

You can frame this in different ways. You can frame it as you learn you you learn over here and then you pass that knowledge over here and and.

B

A

Places, even though maybe this column over here never actually received a certain input, it still learns things that was learns information that was originally obtained over.

C

Here, uh that's one possibility. That's.

A

One way of yeah that's one way of phrasing it you can take that and subtly change it to where it's not so much a learning. It's more of an inference thing uh there. There are different ways of describing this. That's that, but yes,.

C

A

Of the extreme examples they could be completely different sensory modalities, yeah yeah yeah, so that this is the the general problem space where I'm uh of knowledge transfer and uh an observation and I'd like to make is um well if we keep landing on a certain solution or the different flavors of that solution.

A

I'd say the first one, so here I'll describe the diagram here, I'm showing three cortical columns and somehow they're communicating a description of an object, not just an object id, but some actual, like you know, set of parts in their relations to each other being communicated acros like horizontally.

A

We have arrived on that that solution in many different flavors. I think the first one I remember is subutai back years ago doing it with shared feature, location pairs, and this goes back a little bit into the mindset of you go back to the columns paper and others, and we had feature location pairs being represented in layer 4., and if somehow you can make those shared between objects, then the horizontal connections could could do the job. At that point, it might not be layer four anymore, but.

B

A

That was like an original version of this idea shortly after I did something really similar with um I was this was for me uh with displacement cells. Using you know, the the you know the transform between two unique object, spaces and storing a bunch of those um one of the exciting parts of that was that now you could have that, be the thing that you're passing sideways uh and then that would be a way of communicating descriptions of objects horizontally.

A

um I don't think anyone else ever liked that idea, because that particular aspect of displacement cells- one thing about that- was that that would that would mean all these columns would have to have the same grid cell space.

A

They would have to share a good cell space in order to communicate that so uh that wasn't universally liked here, but but that was part of the fbl um more recently um we were talking about graphs a lot and jeff has pointed out that um that the graph can somehow be communicated horizontally in a way that uh yeah, yeah, sure, okay, I see I see pieces of the graph can be communicated horizontal, um so.

D

B

First observation is like.

A

This diagram, I mean, keeps coming back in various forms where we just label things a little bit differently and, um and so to me like, if I just jump ahead like a year or two and think of where, where where this leads. If we go down this road, um I start to think of the cortical column a little differently, um and I started I started thinking I'll. Just like give you kind of like the spoiler the end result and build up to it. I start thinking of it as more.

A

A cortical column is learning a language for representing learning a language for describing objects. It's learning how to take some representation, a description.

D

A

And and um and once.

B

A

Description it it's it's predictive, it's the.

D

Predictive model of an object, it just needs.

A

To be fed a description, maybe it's fed that description from its own memory. Maybe the cortical column is memorized to a particular coffee cup, or maybe it receives it from elsewhere. Maybe it receives it horizontally from other problems. It uses knowledge, that's elsewhere, it's jeff, just put it uh and I start to uh I'll describe these pictures in a second. I start to uh shift my knowledge where the fundamental thing of cortical column's doing maybe this is just like narrative opinion, but like that.

A

Maybe both are important, but in my mind the fundamental thing is the cortical clown learns to okay, I have to introduce the pictures, the analogy I realized, so um I I'm influenced by douglas offstatter here and gerda sherbach, which I've read like two-thirds of uh he uses a record player analogy no.

C

A

It was years ago I read two-thirds of it, we'll see when I get into the final third. So, um by the way it was not. I read parts of girdle etcher and bach, it's not just one of them. It was kind of anyway uh bad joke. The.

A

In it um he likes to use the analogy of a record player and- um and I don't need to go super deep into it, but it kind of stuck in my mind. The notion that, like a record player, is something that takes a description of a song and plays it.

A

The information lies in the record, uh but the record player like what it um what it has been engineered to do it doesn't learn to do, but what it is engineered to do is be able to take a description of a song and play it, and that literally for, like four years, I've been uh three or years or so. This analogy has been in my head. That cortical column takes a description of an object and plays it so to speak. uh It takes a description. It can use it to make predictions.

A

It can use it for another scenario, I'll talk about in a second uh the. So you know the idea of yeah, the the cortical column has an object player. So fundamentally, the cortical column learns to play novel objects, given a description, secondarily importantly, but secondarily in terms of a learning problem, it.

E

A

Those descriptions once you have some.

E

A

Pattern here that describes an object, memorizing it memorizing, a particular coffee cup um is a straightforward problem. The difficult the the advanced problem with solving is learning the language for expressing the objects.

A

F

Great thought so from gb. Wasn't this uh wasn't the argument coming from uh coming from the fact that he was saying this sort of like a turing machine with a you know, with a tape where the description of uh of the problem uh lies in the encoding on the deep.

A

um Maybe he talks about record players in multiple ways.

B

A

F

So, if that's the case then then the tape would also uh it could serve as external memory.

C

That different than what marcus is saying um I mean, is the tape. That's the description.

A

I'm going based on my memory from.

C

A

Ago I mean the big. This goes off topic, though his the big, interesting philosophical question he was asking at the part I'm thinking of is he's comparing record players to jukeboxes and how you're putting in information into it. In one case, you're putting in the true information.

B

A

B

Other case you're pressing a couple.

A

Buttons, and so are you actually putting in the information of a song, you have the spectrum of record players jukeboxes and where the information lies, which is that's the talking point. I remember.

F

From the book, okay.

A

The record player to jukebox spectrum is a really fun idea by the way, but that's not what I'm talking about here. uh So hey I mean that's the full content of what I wanted. The idea I wanted to bring up here, but one one example scenario that a lot of vision scientists think about that. We haven't.

A

We don't talk about this, one that that often but um say you have a particular object that you want to succumb to, and you don't know where it is in your visual field, say: let's, let's use our mental mind, but let's use our our um point of view. Let's.

B

Say the model of that object has been learned in.

A

The past and it happens to be in your phobia- you learned it in accordance yeah, yeah yeah. Sorry, yes, you know exactly what I mean yeah.

C

So yes, your your.

A

C

A

The cortical column that is processing the input from the phobia has learned. This particular object through a particular object, and now what you want to do is use it. You want to say to it well,.

C

You want to inspect it being like, you know where it is no.

A

And meanwhile, it is somewhere in your visual field, but you don't know where. Okay, uh what you can do is, let's just call it a broadcast. You broadcast um a description of this object uh and now the cortical column that um that is, processing input from the periphery receives that message, among others uh and and it detects hey. I see this thing you you give me a description of this object.

A

I know what what I tend to see when an object like that is in my receptive field and then it it like outputs, maybe essentially a motor command. It says like cicada me, essentially uh this.

D

A

Plausible mental model to me for like for one of the things that visual cortex is doing, and this notion of passing around descriptions of objects and living somewhere. Remember sometimes it's memorized. Sometimes it's not memorized. Sometimes it's picked up temporarily and it's passed around uh to me.

E

A

D

A

The thousand brains theory that works for me and I and I wanted to just bring up that that mental model and see if it's, if it if it fits with your mental model, if it's a possibility, if there's something you like.

C

About it don't know.

C

Can I just add some flavor to this yeah so- and I mentioned this recently but I'll bring it up again because I don't think I've never written it down.

C

um You know we talked about, we talked about the description of the object and and then what we're really doing.

C

I made the point the other day that, when we're passing around, we talked about voting in the columns paper. We said: oh we're, voting on the object id and then we said no we're not really voting on the object, we're voting on the object id, but it's posed too right. It's it's like um the pose. Is you know where? Where is it uh and how far away like the direction angle like it to me? Well, actually, actually! Well, no, so you take that pose, meaning it's orientation to me right at the moment.

C

um uh So there's two poses here but right. I should not call post because you want to suppose for something else object id in its orientation. So I don't just see the coffee cup. I see it at a certain angle to me right. It's not, and I have to vote. That's because I'm seeing that, that's what I perceive that's that's what would be voted on. It wouldn't be just the it's just, not just coffee cup out there. It's it's coming up at this particular orientation, um but I also argued the following.

C

I said so what occurred to me is like if I see something like I'm seeing this object, even if I don't know it, I feel like I know some part, I'm seeing something maybe or lip or hold call pick up, but I immediately know how to get my hand to it. I knew I knew how to get there and uh similarly- and I know what I'll feel if I, even if I even if I close my eyes and I reach out- I don't know what to feel like.

C

Similarly, I can be touching something, and I know it's going to look like even before I see it even a novel thing, but I feel something I recognize. Oh there's an edge or a lip, or something like that. I know what it's going to look like if I look at it, so that tells me there's this communication going on instantaneously at that moment right and so what what it seems to be required is what information is going from. My musical portraits my tactical is going to be um it's going to be some feature.

C

It could be. The feature could be the whole cup or could be the lip. um It's that it's at some location relative to the body. It's not relative to my eyes or to my to my um fingers, because that's not very useful, what's useful is to say, if there's some common point, we can both agree to, like my finger column, can know where it is.

C

That these two both can reach. That's why I can reach to it, even though uh you know, I can't see it uh so it's location belts in its body and uh in its orientation uh relative to the body, um and so this could be like. I could be doing this. um So imagine um imagine these are the three things that are that are broadcast.

C

B

C

Relative to the eye or the skin, it has to come to a central thing right. It has to come to a central reference and that's how we that's how I think about it right, I think about like where's, the coffee cup, when I, if I was trying to remember where the coffee cup was. I said my episodic memory, like oh yeah, I picked up that coffee cup this morning. I don't remember where it was. To my hand, I remember where it was my body that was on the table right here next to me.

C

So this this was a ki uh inside terminal. You get a relative to the bottom of the body. Now you can do all the things you just talked about, because this means, um while I'm, if I'm learning a new object and I'm touching it. um Even if I can't see it, I can actually visualize what it is. I can visualize what it would look like as I go through this, and so this was broadcasting.

C

Imagine you have a bunch of columns here and um one column is actually getting real sensory input and the other one isn't this one doesn't get anything, but this information is being transferred back to broadcast all over the place. This column essentially has all the information it needs. This kind of all information needs to build the model.

C

It says: well, there's going to be an edge at some place at this location relative to my body, there's going to be a another curve relative to my body, and I can visually build up this model as I'm learning it here. I could learn it here now. Of course, these are all in reference frames of the body.

C

I have to transfer them back locally to my irish and my finger, so we have to get this sort of clearinghouse to get that done, but if this is what was transferred, then this is what you would this guy could learn it at the same time um and um or um another thing another classic example is we do we always talk about like signing your name with your foot right I mean, I don't know if you like, you can do that and of course, you've never learned deciding, but you can do it and so what's going on here is you're mentally playing back the sequence of movements.

C

um You know like I'm, moving an edge in this direction at some point relative. You know- and I can just put it someplace relative, my body, so I can reach down and touch my hand, but I can also just imagine doing my foot and I go through the sequence of features of positions and orientations, fees and business, and essentially I can draw out the same. If there's some pattern, I'm going to drive here in space, I can draw the same pattern space here um in real time and I've never learned it.

C

So it's not like everything's coming around everywhere. It's not like if I learn to sign my name here, my foot's going to run out of sight, but the information is there. If I, if I would broadcast it, I could learn it or I could do it in real time, like with the sun.

C

So again, another example, if you imagine, if I only learned the coffee cup to with my right hand, and I've never touched it with my left hand, there's two scenarios. One is while I'm touching it with my right hand, I learn it with my left hand. The other side is I don't, um but when I want to do it with my left hand, um I am not as hassle with it, and so what that means is if I've never grasped this thing.

C

With my left hand, if I grasp before, in the left hand, I won't one, one touch, won't be sufficient, but imagine I could say all right. I'm now moving my left hand over the object, and- um and now I can learn the knowledge I have the information over here.

C

So like all my points is that I don't we don't have to learn it in both sides um for this to work, you could learn on one side and then infer on the other side, but it would be slower because there's no knowledge of the entire coffee cup over here, all I'm going to be doing is going okay, a coffee cup has got an edge and a surface and a handle, and I can I can.

C

I can pick those pieces up here, because I've learned over here edges and surfaces, even though I haven't rubbed the entire coffee cup, so I guess what I'm trying to get here and then I continue to be prepared to talk about this. Is we this feature? This object id? um There has to be a some knowledge. Some shared objects between these two, so this guy, this guy may know all about coffee cups.

C

um This guy does not know about coffee cups, but this guy also knows about the edge and the handle, uh and this guy also knows about edges. Something like that. Can't allow the picture, and so you can say well all right. So um I can't I can't infer the entire coffee cup over here, but I can, if I just trace the edges or I can't um you know, I can keep going until there's a common language between the two of these things.

C

So all I'm saying is that it works whether you transfer you learn in both places or you only learn one place and, uh and you pass the components that are shared between the two.

E

In some ways, how does it work if, if you learn different components that are non overlapping between two different columns, how do you infer that it's the same object.

C

I think you have to have the same components. You have to be able to get down to some basic component, um so, for example, um I mean it could get down to as simple as edges, right and curves, um and I'm arguing that both columns here have to know at least some common basic thing and I may have to. I may have to concentrate like when I'm signing my name with my foot.

C

I don't. I have to actually really consciously think about it. I have to say: okay, I'm going to move it up and I'm going to curve and down and under here and up and down, you really have.

B

C

About it, when you're signing with your right, with your normal hand, you don't really have to think about it, so but the foot and if the foot never knew, if I could never imagine, drawing a line with my foot or how to move my foot in a straight line or a curved line.

D

C

I could do this.

C

Another example might be what if what if something was determined purely by I'm discussing them, I'm making this up what if something was really determined by colors only uh and this was and then this is the tactical column which can't do that um and then the argument here that I couldn't well, I couldn't I couldn't. I can't imagine what it would feel like right.

C

I could say: oh there's these different colors at different positions uh relative to my body, but I can't I can't imagine what that feel like, because there's no common language about color between these two, so it doesn't always work right um same with sound. I can I can. I can learn a melody or a sound of a truck or I don't know I'm not my point is it doesn't always work. There has to be some common component that goes between these two. I can say.

C

There's this feature at this location relative to your body and as long as.

B

They can agree on that. Then.

C

They can do this, but the features can't be shared and then they.

E

A

So one one good point: you've brought up you made jeff in this case. uh The these. These descriptions of objects um are typically allocentric, they're they're, not relative to the body they're right here, I'll say this differently. The thing you want to learn you you, if, if you're gonna memorize something you probably want it to be allocated, you probably don't want it to be relative to the body. So you bring up an interesting idea that the thing being you know that the message is being passed, sideways, maybe they're always relative to the body. Well,.

C

I was thinking right, I was thinking you know this and you know more about this literature than I do. There's there's like cortical column and there's cortical columns. These are all collagen columns and then there's a bunch of literature about like something called the striatum um which looks kind of like cortex. But it's not it's sort of stuck in the of the brain and there's a lot of literature about how information goes in the stride and back up again you're the first person to point this out. I think.

A

But I kind of I kind of side with what you're saying: let's try it I'm become I'm becoming more supportive.

C

Maybe maybe maybe you should just be cautious, but I mean I think it's just right and I think you're right. That's that's what the literature says. So so maybe you said the question: um that's right: suicide. It.

A

Is this writing is associated.

C

With reference frame transforms and- and you know, I have.

B

To I have a whole file.

C

Of papers on, I have to go, read them again, but um the idea here would be um that you know going through this sort of reference frame. Reference to the body is.

B

Kind of an odd thing, because.

C

Locally, we don't really want that locally. We want to you, know, change the fingers with the eyes or whatever right um uh in the object. So this would.

B

Be like okay, this is this.

C

Might be part of this whole communications method here that we broadcast you know um we broadcast through here um to other places it doesn't mean there still isn't local communications as well. It just means that, for this knowledge transfer component there has to be a there has to be a pathway through the body's reference fact. Otherwise it doesn't work, but it's so obvious now to me that you know, even when you hear something or you feel something or you visualize something you visualize it in some position relative to the body.

C

That seems to be your your mental concept, um not that you can think of otherwise. I can also imagine, what's it relative to my elbow or something like that, but but normally you just say oh yeah. This is off to here and I can and that's why I can all my senses. Can I can't even imagine I.

D

Can even attend to hearing something.

B

C

This location- I can be standing here, so I'm going to listen for something down here. You can just mentally do that right now. You can feel like. Oh yeah, I'm going to listen to that spot.

C

So um so anyway, there might be this um clearing house of uh transfers uh to get there. It doesn't get rid of the local voting. It doesn't mean you still do local. I do, I think, I'm sure, but this could be like okay. This could be the transfer knowledge situation.

A

Generally should be allocentric and these pictures when we talk about transferring location barriers, the displacements, the graphs, those are usually allocentric, but but your point, which totally might be right, maybe all of the transfer things are uh and the body's reference. I.

C

Just see that that came out because it was also there was this uh introspection observation about that. Also it solved the problem of you kind of have to have this to do um to do the to solve all the knowledge transfer problems yeah. It's also obviously important.

A

For voting as well or if you can vote on these things, if you can vote on these things,.

C

Yeah you even brought up early with the with the multiple fingers where you said you know it's not each finger.

C

uh Each finger is not just tracking its own location, which makes it more like a bad features problem, but he said they know where they're, where they are relative to each other yeah, um and uh when I agree with that, I put that in the book. In fact um talking about the people dropping into a town, um so that would be that's sort of like um something like this. It might be required to provide the common basis by which everyone can agree. Yeah, like I'm, you know I'm at this location.

C

Where else would they call the cup I might be at this location, but where are we relative to each other? um If I don't know the relative positions of my fingers here, then um then I just have a bag of features, um and that is insufficient.

E

So what layers do these striatum connections.

C

Go to and from uh you.

A

C

Great question I don't know, I have to go, look at it again.

A

My instinct is deeper layers, but I I'm not going to give a straight answer to this. I do.

C

I know I remember one okay, this is a really detailed, neuroscientific. Okay, so I remember one now um in general, if you talk to a neuroanatomist- and you say, what's in layer, five excuse me they'll say: oh there's, two types of cells in layer, five, these two types of paranormal cells, big ones, are little. The big ones are the motor output cells.

C

These are the motor and then there's these smaller ones, with just regular parameters and these project um long distances. These are one of the voting neurons. I mentioned the other day because there's voting neurons in layers two three and there's these long distance ones like thought and you see, then we read a paper once so that was this. Is this is standard dogma big? This is called the layer, five intrinsically bursting cell, that's because it makes these little bursts and these pt ground little tracks uh parallel track is another name for classic certain nervous line.

C

Two multiple names just same thing um um and sometimes.

B

They're called five and five b, but in some animals the reverse, you really have.

C

To pay attention to this, but there's.

B

Another one and big ones and the.

C

Big ones, basically, you have the motor up for the cortex and little ones project long distances, and we would say they wrote it. Then we read a paper where someone said: ha ha there's actually a third type of layer of lifestyle.

C

Remember this! I don't think you guys remember this one. He said that okay, well, there's the big big motor one and then there's two flavors of small ones, and only difference between these. These two look identical except for one of them, so they look at that and call me and they project the same ways. He says, but one of them has a little extra projection off to where so, let's try him um and they made the case that there were two flavors of these small guys.

C

Does anyone else remember this paper yeah? um It was like one of these like you could just get buried and there were signs of crap and- um and this is one of those papers you could just. I can't deal with this right now.

C

But um this this particular science is arguable, there's a second class of these small layer, five cells- and that begs the question. Well, maybe there's two different types of voting going on. um We could probably find this paper. We did review it. We did a paper. Well, I really I at least talked about it.

C

One time um and um one could argue- maybe there's two different types of voting going on here, and one of them is voting on the location relative to the body and maybe one of the building, a location relative to the sensor, or something like something like that. um So that was another uh data point. um Just where did the dopaminergic neurons come from? Is that also striking.

D

By the way, just to.

C

So you don't know it, it just means lower it's stuff.

B

C

There's a whole bunch of things that are called the basic angular, there's just a whole bunch of little things. They just group them all together and called basic angular.

C

You know those things will project broadly in layer, one or something like that, and so it's more of a global learning signal than a specific um information system you can tell or cortex people.

C

Why? Because someone would disagree.

A

No, no I'm just saying that, like for some people, dopaminergic neurons are the center of their universe yeah when they study the brain. So it's just fun to see us have to ask: where are those again.

C

Well, because an experimentalist is something they can deal with, like, especially with an animal like a rat that can provide motivations for that that's hungry. These rats in these experiments are almost always hungry and thirsty.

C

You know like they don't remind you all the time, but their rats are not very happy and um they're really thirsty, and they really really wanted something to drink, um and so they can then do these experiments on motivation and reward and so on. So they do all these motivation things with dopamine, but that's largely because it's something they can deal with. They can't ask the rat. What are you thinking right now? You know what do you what's your perception of something that they can't deal with so anyway? Yeah?

C

So you know, don't mean project typically broadly like layer, one or something you know, and then it just infects everything about. It's learning how to learn.

A

So it's useful for me to just check if uh I'm gonna try, I'm gonna see if I can make the most radical version of this claim and see if you still agree with it, uh or rather I'm gonna, I'm gonna bring up. Why I why I feel uh why I call this a slightly different flavor of a thousand brands. Theory: that's compatible with it, and it's like when I think of a cortical column, processing input from your periphery, some somewhere over here off to the side.

A

um The idea that that cortical column is learning entire models of objects. um I don't disagree with it, but it's not the narrative that works. For me. The narrative that works for me is that what is fundamentally doing is learning. It is learning to represent entire models of objects, and that is this fundamental thing. The idea that every single, your cortical columns is like a little straw, that is, that is each learning the objects. uh For me, it's like almost they're, not quite how I think of it. I think of it as more.

A

It's learning to describe them. The memory might not live over there, like especially like way off in the periphery. It might not really.

C

Be memorizing I mean this is one of the perfect vision because yeah I agree, I mean it's like the eye is like two things right, there's a phobia where everything seems to be all the sort of learning and recognition seems to occur through movement, and then you got the periphery which is dominated by uh you, know the the magma cellular sort of cells which sort of detect motion right. It's almost like one could argue that the outside of the eye is there just for just for motion detection. This could be an argument um that and.

B

That you're really not learning much out there. It's just like.

C

You know it's those slow bits, I'm not sure. If that's true, I don't really know, um and so I'm open to both suggestions. I hope the suggestion says: hey these columns are represented, the outside of your periphery are winning things but they're, not very good. If they're, very fuzzy or maybe they're not running much at all, I mean that's what you're I don't know.

D

Could it be that those cons are just learning much larger things? They.

C

Could they could, but it's so poor that no matter what you do, you just can't learn much out there right. It's just. It's really really impoverished data.

D

It's like looking through the fuzziest filter. You can imagine so is it? Is it actually impoverished, or is it just over? It's still, the same number of inputs is just over a larger scale. um It's not like! Suddenly it's it's. You know.

C

I can't see questions, let's say: let's say we want to see, let's say we're, trying to look at a coffee cup right and- um and I can look at the straw in my center of my vision- okay good now. What? If I give you a really big straw of the masada method, can you still recognize the coffee cup.

D

If it was uh ten times bigger, yeah one sure, I'm not sure.

C

D

I mean I mean someone must have done these experiments. Probably I'm.

C

D

Couldn't recognize letters.

C

You know, I think yeah.

D

I mean the number of raw inputs coming into the cortical columns. Here is probably the same as the number of inputs coming in here.

C

It's just over what are they? Are they the magnitude of the problem, because that really changes a lot right yeah? So it's going to be dominated more by the motion sensors out here, so one can argue that maybe it's going to learn them. You know maybe right. I can sort of see this big letters over here yeah. So I'm not sure your distinction there, marcus you're, saying that um do you think? Maybe we don't learn these things.

A

Or that's the general tendency, the general tendency, the scenario you were just bringing up is probably most that knowledge actually lived more in your foveal cortical columns and to the extent that you were predicting what's over there or to the extent that you could find the coffee cup in your visual field. The other scenario I brought up in this box.

A

It often involved a knowledge transfer like an on-the-fly information being sent over there yeah you.

C

Know it could be the equivalent in touch would be you know um I I wouldn't have things to touch with my fingertips, but you know. Could I recognize them with the back of my forearm right? I don't think I have a model of the coffee cup in the back of my forearm right. um I don't think I don't think I ever you know. I don't think it exists there, um but your argument is, I could infer it.

C

It's like I'm doing the knowledge transfer. Now I'm trying to imagine what these as I touch this I'm trying to imagine some draw a little visual, almost imagery of what it is. That's out. There is my move. I can write the body around like this, um so I think that's maybe right you know. So it's not again. We've always said every column in the quarter. Cartridge doesn't learn models. Maybe some some columns are not very good at learning models at all.

C

Maybe that's your argument and then we're just using them for the the technology transfer inference problem.

A

Is that right? It's like yeah, basically yeah, but I can also say some flavor of what subatomi was saying that, like they do, learn something your periphery.

D

Does learn things, but it learns different things.

C

Yeah yeah, I think there are big things that are important yeah, so I I noticed the other day I was in bed and I was trying to it's really you're lying in the dark thing. You have this really rich sense of where the blankets are and where my life is and and and there's only these little touches on your body, and you somehow even know like you just know what those touches mean, though no there's an extra fold in the big blanket. You know they just know these things like look at this touching image.

C

So one argument would be that you know yeah. The different parts of my body have learned like things like what it feels like to be on a blindfold like that, but they don't learn things like coffee cups because they never touch the coffee cup, but they do learn blankets.

C

um One could argue that if I really, if I had to, if I lost my forearm due to an accident, then I could probably I'd probably get really good at sensing things with money.

C

um So I think that's. I think I would rather say that the different columns learn different things. um They can all be subscripted uh into performing action like I could I can infer with my forearm. I can draw my name with my toe, but I'm not generally gonna learn that so just because I learned something with my fingers doesn't mean everyone's gonna learn it, um but everyone could imagine it right as I'm as I'm touching it or as I'm mentally walking through a model.

C

I can broadcast it to other people to say well, if I was in that location. This is what I would be feeling.

C

This same mechanism is the base of the language um that you've got these language areas of the brain, never just cortical columns like every other yeah right and um they are able to, of course, move your voice box and music.

C

But what are they? What do they want to say right if I wanted to what they want basically to describe is, what's being broadcast around the rest of the brain. You know it would.

D

C

Well, marcus is off to my right here and I know jeremy's on the screen over there, because I just saw him- and um you know my coffee cup, uh and so I can describe basically what what I'm imagining or seeing over.

B

Here all you need to do is say.

C

You know what is the object? What's its orientation? Where is it relative to me, and almost that's like the core of almost all language is, is recreating whatever um um mental imagery or actual sensory input is occurring elsewhere. If I want to generate.

B

It, and even if I'm talking about.

C

Something I'm just thinking.

B

About mentally, it's the same basic idea.

C

There'll be columns representing some sort of concept and they're going through their imaginations, but it's still being broadcast as long as as long as I can associate these these these objects and their orientations and their positions all to the body, then this guy can describe it and simply, if I get if I get auditory input- um and I recreate what's going on in your head and pass around the rest of my brain, I can imagine what you're saying you can say. Yes, jeff was just describing the coffee cup on the table.

C

I can imagine the coffee cup on the table and and then there's a car out front that had an accident. Yes, I can imagine that um don't have.

D

A relative to the body part was.

C

D

How do you do that? If you don't have one elective to the body well,.

C

It's a good question. I don't know, I mean.

D

um Describing something that is not here.

C

Like well, let's go through an example, let's say eight.

D

One plus one equals two how'd. You call that.

C

C

Let's pick something I don't know so that you're trying to describe something that I don't know um well, okay, the relative to the body part's an interesting question. I don't know I don't have the answers to that in in in any um anyway. I'm just trying to imagine like describing something mathematically.

C

uh um If I would describe a mathematical problem to you and all we could do is talk. um How would I do that? I would probably use visual language to do that. Even if I was describing this equation or I would um I don't have to say weight as well to my part, you can imagine it anywhere. You might imagine in front of you say louis.

C

You know I'm dealing with this interval right now and it's between you know blah blah, blah blah and we're differentiating all of this, and I mean you'd, probably mentally, build up this image in your head. With that same thing, um would it be? I don't have to say where it develops in my body?

C

um So that's an interesting question. I don't think it's a fundamental flaw in the idea. You know I'm still describing something that you're going to make a visualization of my head, I'm just like for the.

D

For the location that goes to that.

C

Yeah, I don't know, I don't know what would happen there uh interesting question.

C

You know it's, I don't I'm not worried about it.

C

The other thing going back to something you said earlier marcus. um This is sort of uh uh somewhat related to this. You could pass you could vote. You could pass around something something that you're looking for right, but here's something I'm looking for and- um and I think the.

A

Same mechanism might work uh you might.

C

A

Know basically, you.

C

A

What orientation to expect it to.

C

Be yeah, you don't know where it is. The point is that all these guys could be biased to find that thing yeah. You know where's waldo, you know um yeah. I know if I it would be interesting if you played, whereas waldo you turned the body of all those sideways, so it starts go up and down. Would it be hard harder?

C

It might be because, like if you look at the walls, you'll probably always grow, I don't know I mean I thought I thought this. I thought this is really interesting about language. It's language is always really weird to me because in the cortex the idea is that somehow information flows hierarchically through the cortex, then it gets the language, the language areas and it flows back down to do things, and it was like how does that work? Why is it gonna seem different? It's like you know, let's, how does language abstract something from the visual scene?

C

And you know all these weird things. This solves that very nicely. This is languages. It doesn't notice that the columns are getting the same descriptions and they can now turn those descriptions into a physical manifestation of written or spoken language or they can um or they could. You could feed in some description to them as if they were sense organs. If I'm I'm a parsing language- and I understand it, then you can pass that as it is just instant itself and then these guys can visualize it.

C

So it just makes language areas just like uh every other area. um In that regard, they they have a sensory input. In some sense, it's the words and they have a sentiment, an alpha which is creating words and those words are describing the same objects that we're describing up here and passing this thing around, so that I thought we ever want to do language stuff. That was like.

B

I'm mostly trying to understand yeah like just the framing of the problem. If, if you're trying to look for like a common language, descriptor that works between all of all of the columns or or like a way that the columns can each learn a common language between each other. But it can be different between the different.

C

If I think of it, as.

B

C

I would say: well the location though, to the body would be common of all the columns like since hearing touch and vision. The orientation of the object is probably uh common to all the columns and then the actual features that exists are common to subsets of columns um and but it wouldn't. I don't think, you'd, learn it on a column-by-column basis.

D

I mean part of what's going on, is the column has to get votes from a bunch of other columns and figure out? What's most consistent with the evidence, then you would, it would seems like it's advantageous that the same language, otherwise it's yeah.

C

It wouldn't be sound like, but you can't always have the same language. My point: there are sensory inputs that can't be shared. There's certain objects that cannot be shared.

C

I can't visualize a melody unless you're really good at music. Well,.

E

um In in synesthesia, you can right.

C

Yeah, that's that's a synesthesia, the best theory about synesthesia. Is it's a it's a a wiring mistake in the brain. It's not! Yes!.

E

I agree, but so.

C

So we can ignore that, because it's really not useful, it doesn't really help you much at all. It's just it's just a spillover wiring effect between two columns that are adjacent to each other that actually represent different sensory modalities.

D

What about something like these multi-agent scenarios, where you have little simple things that are going around they're, trying to do something common and so everyone's sensing, something different but they're, trying to recognize something common or do some common goal? Well,.

C

I think in some sense that's that's kind of what's going on in the courtroom.

D

Yeah yeah, so we you're saying we don't care. What's inside the column, uh it's just that they're communicating on something. That's common to all of them and inferring things from partial information, yeah there's you could imagine some multi-agent or some multi.

C

Yeah, well, I could argue that my body is a multi-agent intelligence system right right and so every patch of my skin, the retina and the parts of coconut oils parts of them are moving together, partially moving independently and uh but they're all communicating with each other, and so the same basic principle could apply over broad areas, but I mean like I could have multiple sensors moving around the room independently of each other.

C

If we wouldn't work much, if some sensors were you know, physically remote and couldn't even be sensing, the same thing like it. Would that kind of thing wouldn't help? If you have some sensors and regular city and substances.

C

It seems like they have to be able to somehow at least have they can imagine attending to the same physical location. That seems to be required attending the same physical.