Numenta Numenta Research Meetings, 22 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: "Learning Physical Graph Representations from Visual Scenes" Paper Review - December 22, 2021

Description

Numenta Research Intern Abhiram Iyer presents the paper “Learning Physical Graph Representations from Visual Scenes” by D. Bear et al.

He first gives some context and an overview of physical scene graphs. He then explains the pipeline of how these graphs are built in the deep learning system, starting with 1. feature extraction 2. graph pooling 3. graph vectorization, and 4. graph construction. Lastly, he goes through the results from the paper, the caveats, and his main takeaways.

Paper: https://proceedings.neurips.cc/paper/2020/file/4324e8d0d37b110ee1a4f1633ac52df5-Paper.pdf
- - - - -
Numenta is leading the new era of machine intelligence. Our deep experience in theoretical neuroscience research has led to tremendous discoveries on how the brain works. We have developed a framework called the Thousand Brains Theory of Intelligence that will be fundamental to advancing the state of artificial intelligence and machine learning. By applying this theory to existing deep learning systems, we are addressing today’s bottlenecks while enabling tomorrow’s applications.

Subscribe to our News Digest for the latest news about neuroscience and artificial intelligence:
https://numenta.com/news-digest/

Subscribe to our Newsletter for the latest Numenta updates:
https://tinyurl.com/NumentaNewsletter

Our Social Media:
https://twitter.com/Numenta
https://www.facebook.com/OfficialNumenta
https://www.linkedin.com/company/numenta

Our Open Source Resources:
https://github.com/numenta
https://discourse.numenta.org/

Our Website:
https://numenta.com/

A

So the paper is titled learning physical graph representations from visual scenes. The entire idea here is: can we take a look at a scene and try to build some kind of hierarchical graph from that scene?

A

There's a lot of technical jargon in this paper, some of which I've included some which I've completely abstracted, hopefully to benefit the clarity of the presentation.

A

So feel free to ask questions along the way.

B

You can hit the arrow or spacebar just did you. Click.

A

On the thing, I guess I'll have to control it, okay, um so the basic idea behind why this paper exists is because cnns fundamentally do not learn structured representations of a particular scene. So there's no way to to understand how a particular object works with other objects. There's no way to understand what are the components of a specific object.

A

I think these are properties that we're all pretty familiar with. The paper highlights that humans effectively group scenes into object-centric representations about various objects in the scene. Their poses their characteristics, such as their physical properties and their.

C

Relationship to other objects, so these, like these words, are these words: are you taking from their paper or these words you you're saying, because this is how we talk about it too. So I'm just curious. Are they using these? These terms.

A

Yeah, so the I I intermix, my my words with with some.

C

A

um That that sentence is um from their paper.

C

A

Yeah, um so their idea is to introduce this this notion of what is called a physical scene. Graph, like I mentioned, that's basically a hierarchical, hierarchical graph, particular scene, and this is purely a deep learning system to do this, and I think the key thing that they highlight in this paper is that they learn this by self-supervision there's.

A

So there's no need for a ground truth, labeling of the scene and the particular objects in that scene, and they test this basically on very simple environments like gibson, which is similar to the facebook habitat environment that we're using right now. But it's it's not interactive um and primitives, which is effectively just like a 3d room with some primitive shapes.

C

So are they? Are they working with 3d basic data, or are they working with images and extracting graphs from the images.

A

So the environments are 3d but they're, technically working off of 2d frames in movies that they process so.

C

So they take a 2d frame but they're also taking time versions of it. Yes, as if the sensor as if the viewer is moving through the space, yes.

A

C

All right, so it isn't um okay, so it's! It is a in some sense. A sensory motor system.

B

C

A

B

I guess you'll go into this in a bit, but do they actually control the movements or is it just passive, but the world is it's it's totally. It's positive in the sense.

A

That I think these are fixed movements in that environment. Okay, um I think uh the the environment is not interactive at all, so it's um the scene, everything all the objects in that scene are stationary.

C

Yeah but but the viewer is moving through the scene. Yes, so um that's the key, that's a key differentiator and then suitable. I think what I was asking well, but they they're not taking advantage of the actual, not knowledge of the movement, perhaps they're, just they're just looking at how things change.

A

Yeah I mean if I interpret that you know the from the network standpoint: it's like they're, seeing video sequences yeah yeah, they have no, they have no control or anything of them. So it's not really sensory motor. It's more like analyzing video.

C

Yeah, but it is, it is in some sense, uh potentially able to um uh it'd, be like us watching a video of someone moving through a maze or a game right we can you can.

C

We can, even though we're not directing behavior, we can use motion clues uh that exists in the in the video to determine depth and and direction of movement. We can. We can infer movement from changing video.

A

C

That's right, I just.

A

Wouldn't call it sensory motor.

C

Okay, but it's getting closer, I mean here we're talking about basically learning three-dimensional graphs of complex uh hierarchical scenes uh by movement, so it's getting closer.

A

Yeah yeah and there's a there's a fair amount of there's some other work along these lines too. But that's one of the reasons I think we want to.

B

Talk about it, yeah.

A

I'm gonna highlight some caveats later in the presentation, but I think I should say a few now that there are a lot of deep learning tricks and uh strategies that they use to make. This work, um like, I said, I've. I've abstracted the ones that I think that are that are not going to help with the clarity of understanding this system. um I think we should still keep that in mind that this still requires a bunch of trading data and all that stuff to work.

A

So the basics of of how they build this graph is everything is built bottom up. So nodes are basically, they say at the top of the hierarchy at the top of the hierarchy, but it's considered um the the nodes that you start off with, and these are the larger groupings like whole objects in this particular scene and nodes towards the bottom of the hierarchy. But this is technically nodes that you you build over.

A

Time are smaller groups, like particular parts of nodule, and the edges are actually how various objects are held together, so this graph is very much like a hypergraph, so a single edge can join any number of vertices and, for example, if I had specific nodes at the bottom of a hierarchy that described various parts um of one single object, the edge that holds them together describes how they're held together.

A

um It's worth it's it's very. What.

C

What does that mean? What do you mean by hyper graph how's that different than just a graph.

A

So a regular graph, um an edge, only connects two vertices. A hyper graph. um An edge can connect any number of vertices.

C

I guess I'm not understanding how that's possible uh if a vertice is a uh a feature or an object in the scene. How can I have? um Are you saying that could be have a relative position to many other things is that it be an edge? An edge can only connect two things right. How can this, how can a single edge connect more than two things.

A

It's uh it's useful to visualize this knot as a as an edge as like a line between two vertices, but rather as like a a blob and in that blob are many vertices. So that's like what what's considered an edge.

C

Okay, well then, that seems like that's not really a graph, then.

A

It's it's still a graph in the sense that um I'll get into more about how um hierarchies hierarchies are built inside this field inside this graph. So it's so rough, but it's not like your traditional edge. There's a single edge doesn't connect only two vertices.

A

So in a very simple example, if I guess it's not a graph, it's a hyper graph yeah. I definitely.

C

Should? Okay, that's what you mean by hypergraph! That's why I first started saying what the hypograph the hypergraph is like. Oh an edge, really doesn't connect to things. It's sort of some other concept. It's a blob to blob connection. Well, that's! Okay!.

B

A

I can we think of this.

B

A

Clustering, a bunch of uh vertices together and and then just connecting something some other cluster to some other cluster yeah.

C

I mean in some sense you can say the relationship of the coffee cup to the table is is an edge, but the coffee cup itself is composed of many nodes or many. uh You know components and edges themselves the table. So if you want to think of, if, if that's what you mean, then that's fine, but I would still say you know the way.

C

I've always viewed that as a coffee cup as a single object in one instance, but it can also be decomposed into others um that maybe that's what they're saying here yeah and that's that's yeah. So then sometimes it's maybe like there's multiple. You know, I always read it as multiple graphs there's a graph with a copy of the graph of the table, and then I can say well take that entity and make it a single node, um which is a hierarchical graph, but I wouldn't call it like still in that.

C

In that sense, every edge still connects two things, but all right, we don't need to beat this up more, but I'm just trying to understand it.

A

It's very important to note that these graphs are spatially registered, so every node in the graph is tied to a specific location in the image from which it's derived and the node attributes basically describe the physical characteristics of that object. So the position of that object, the shape and the appearance, the color and stuff like that.

C

Just to be clear, this is a location in the image, not a location in 3d space.

A

Yeah, it's a location in that image frame.

C

Okay, that's that's a bit odd.

A

So the uh I think, as I as I explained, this it'll become more clear, but this is how they refer back to the original um data that they're collecting from your from your camera. This is the only way that they they have like some physical mapping between my graph, the nodes in my graph and what corresponds to the series of frames that I'm looking at in the uh the data, the raw data that I'm collecting.

A

A

A a bunch of text here, I'll I'll go through one by one. um The vertices in this graph are arranged in hierarchies like I mentioned, um and every single um set of they call it. A level is denoted by v of l, and v of l essentially contains all the vertices. In a specific level. L the edges between vertices at level, l and l, plus one- um are what denotes this hierarchy, um and this is what they call a child to parent edge, and this represents basically part-to-whole relationships.

A

You build these vertices as you explore a particular scene, more and more so, if my camera, maybe zoomed in on a particular ball or a coffee cup in this instance, you would represent new things in a new hierarchy or a new level, and you would connect the old level to your new one using this child of aaron edge, and this is supposed to become more descriptive as you go along.

A

There are also edges between vertices in a particular level and those are called within level edges, and they represent some abstract relationships between the objects or importance in them in that level.

A

In this specific implementation, in this paper, they only describe the physical connections between um between objects, in that one uh they could encode various different things like how one object would potentially interact with another object using some physics, um but that's not what happens in this paper. This is purely how um one object is physically connected to another one.

C

Is it physical connections or just physical relations? I mean, is this yeah.

A

I think in so they don't have to touch it yeah, they don't have to be touching. They don't have to be touching, but um there's this very ambiguous case where let's say I have like a coffee cup, and I have it in front of a background and the movie that I'm looking at is in front of both of those things from the viewer's perspective or from the camera's perspective, it's hard to understand. What's touching what, um but that's how they they do this in their case, where they consider the coffee cup.

A

In the background to be separate, because the coffee cup follows some very similar characteristics along all different parts of that same cup, and the background is different- I don't know how they stress tested this in the sense that what if I looked at a background that looks identical to the coffee cup at that angle, I don't know how that's how that's done. I.

C

Guess I'm still, it's still confusing to me we're talking about edges now at a particular level. So this would be like the coffee cup relative to the to the stapler type of thing, and then you say physical connections, the edges represent physical connections and you've emphasized that. So um I would expect the edges to represent the physical relationship between these, like you know, they're separated by certain distance, or something like that.

C

So I'm just I'm just confused by the language of physical connections.

A

In the case of the the coffee cup in the stapler, I don't think that there is.

C

Any sort of physical connection between the two would the what would the edge still represent. Is that what you mean, but because that's what this edge, these edges would represent right.

A

Would the edge say: oh, it's three centimeters to the left of the coffee pot. Nothing like that! No, no! No.

B

So, in that case they wouldn't even there would be no physical. There would be no edge between.

A

The stapler there is yeah based off of how they've described it, there's no edge between the two.

C

um So is this then different parts of the coffee cup.

A

Basically, basically so um I I'm starting to think of this as like the edge represents contiguous parts of a particular object. It's.

B

Not it's not equipped.

A

To describe um displacement between two objects.

C

Yeah, I guess, and the way I thought about this is really isn't really a distinction between those two. In some sense, it's like you can just take any two things, whether it's part of the same thing or it's part of a scene, and you can say what what is an edge would represent the relationship between. I don't want to beat this up again, but you've emphasized this term physical connections and it's still a bit confusing to me, but maybe we shouldn't spend more time on it.

A

um Yeah, I can, I can come back to this um if it's not clear, hopefully by the end of the presentation, um but they also determine something called attribute vectors.

A

These attribute vectors are are very simple, there's vectors that describe for every vertex in a particular level, um whatever the physical properties of event vertex, um and these are purely just uh what's the shape. What's the appearance, what's the color and in some cases they manually provide information like what's the physical normal for for that service, uh for the option that they're looking at- and this is not something that's that's uh derived from from just looking at the scene. This is something that they already have.

A

Knowledge of, um I think that'll become more clear as I go through some examples.

C

That's right, that's very similar. The way we've been thinking about um uh nodes uh being like you would uh like a circus would have a sort of a normal component to it. Are they using the word? Are they using the word vertex in exactly the same way? We use the word nodes or marcus uses the words nodes.

A

They use vertices and nodes interchangeably and I think I think it means the same thing.

C

Okay, it's interesting question for our language. We've been using the word node and these people using the word vertex. I don't know if one is more understood, generally understood or better than the other one, I'm just questioning whether we're using the right language.

C

I, like the word node to me. That was something I understood, but for text means the same thing. I think.

A

I I think vertex is more specific to graphs, whereas those can be more generic.

C

Yeah, I like the node, because because we assign a lot of different things to a node, and so are they, um but you know vertex, sort of suggests a a physical part of a graph, as opposed to like a a a place where you can put a bunch of things.

C

So all right, we'll stick with nodes for now, just curious.

A

There's also something called the spatiotemporal registration, um and this is also a set that basically is tied to any given level of this hierarchy, and this links each vertex or node to a given subset of pixels in something that they call the base tensor, and this is the thing that I was describing in the previous slide. The base tensor is effectively how they have this physical mapping between um the nodes and vertices in this graph, as they build to the specific series of image frames um in the raw data.

A

This is the only mapping that that really exists. um This base tensor is of size um t by height by width by c I'll describe what that means. T is the number of image frames, so we can think of this as like. A movie um and c is the number of attribute vectors in that particular level.

A

So, as you could possibly guess, f represents this base tensor and this base tensor this full base. Tensor represents the ground level node from which the entire graph is built.

B

A

To that point, uh so I mean this is like the if it's built bottom up. This is the starting node for everything. um This is the full scene, um and this is all the attribute vectors at that very very um at that first level. uh This describes all the attributes of all the objects in that particular scene.

A

So if you're like at the.

B

Well, if it links each vertex to a given subset. So if you are at the coffee cup vertex, but you can, then you know all the pixels that belong to the copycat through the through history and through this video is that it's like a segmentation, yeah, basically yeah, but that that's that's taking it. One.

A

Step further, this is if um what you're saying is absolutely correct, but that's built from the ground level. Note there's only one single node that you start out with and that node is effectively the entire scene throughout all the time that you're looking at it from there. You you make it more granular. You would look at maybe a specific node and that node is the coffee cup and you would see how the the original base tensor relates to nodes as you as you keep going up now.

A

Okay, um you would maybe start off with the coffee cup. Then you go into the coffee cups handle and you go into the colors of the coffee cup handle and you just keep going further further.

C

It seems to me this technique, and maybe I'm jumping ahead, but it's going to be very specific to this. This scene, that is knowledge of the coffee cup, wouldn't be transferable to another scene, in some sense.

A

Yeah, so this is a this is something that they actually bring up in the uh in the paper where they say that, even though this one seems like, apparently the the system generalizes pretty well to um to similar scenes with similar objects that are not identical to the ones you trade on.

A

I uh maybe I can I I can show you guys some results towards the end. I don't have them in this presentation, but I can I can pull them all on paper.

A

I'm also curious about how that particular thing works.

C

A

So they have this very, very big pipeline for how this works and I've tried my best to separate them out into their individual components. There's a few parts of this pipeline, the first.

C

One I have I have to interrupt. I have to step out for a minute. You guys keep going and I'll come back as soon as I can. Okay sure.

A

How long roughly, do you think you'd be.

C

I don't know, but this guy's contractor's here and he's. I just have to answer some questions, hopefully not more than a few minutes. Just keep going.

A

You might want to.

A

So this is the first stage of their pipeline. This is feature extraction, and this is given the raw scene or the raw series of images in that scene. How do I process them? A little bit of this convolutional recurrent neural net um that performs feature extraction from the series of images. What they effectively describe is every single layer here locally has recurrent layers that process the the series of frames and then across the entire architecture.

A

They have some kind of skip connections to preserve the more granular features that you learn towards the end versus the more raw features that you have in the beginning. This is, um I spoke to the author and to understand why they did this, and he said that this was basically a manual design decision just to make sure that they have the relevant features um throughout the entire network, so that granular features aren't lost.

A

Raw data features are lost here, but what you see there on the top left is sort of what's: output from the the conf rnn towards the end, um it's just a a representation of the raw data which is right below it. I've heard of something called an rcm before is this different from that? I think this is different.

A

uh This is the.

B

Maybe crm slow, but can we just take us back to understand like what these uh recurring connections are doing like what actual operations are being implemented? I don't know like the specifics that they have like an insight in the top there that describes each local record layer. All I know is that this is supposed to process the image.

A

Screens together, so if I have a series of image frames that describe how I'm moving about that scene in the bottom left, this would this would be able to convert that into one shot. So I'm passing it in over time and that's that's converting information over time I mean. Certainly some level looks like layers in the cortical column.

A

Based off of the the how they've.

B

um uh Listened to the names of coming, it's an lstm, I can't be sure. uh Okay, I'm just kidding the backward arrows adding back in or what actually that's.

A

B

Same connection yeah, is it simply adding back in or is it doing some other operation integrating with the uh the earlier layer? It's just adding it back in by.

A

Virtue of the skip connection, you don't want it to pass through any.

B

Operations, okay, so you've got a smaller resolution there. uh What's the expansion operation.

A

Yeah so right there, there is one fixed transformation that that scales it up. That's the only one, so you you're actually changing the features that so from fc6, for instance, when you're passing it back to con one. There is some kind of up-sampling operation, that's happening, but that's a fixed upside for an operation.

A

It's not something that you're not changing the features that you've uh learned in fc6, to put it back to you, but if it's just like an image scale or when you say yourself, okay, all right all right! Here's the connections go backwards through the now yeah. Is that the same? That's not the recurrent connection.

A

The current connections happen, which within each uh each block. Well, it's still a it.

D

Is a type of recurrency right.

A

Yeah is it the type of requirements yeah, but these are, I guess I guess when I, when I hear a recurrence, I think of the actual recurrent layers of the lstm or the the gru unit, whatever they're using inside each layer. These skip connections.

A

Actually, you guys are right, you guys are right, because, if um they're just a terrible.

B

There's spatial yeah yeah.

D

I feel like it's closer to re, calling it a recurrent connection than a skip connection like if I hear skip connection, I assume skipping a layer forward and not going backwards, so they describe it as skip connections.

A

In the paper, I'm I'm honestly wondering if the diagram means that they're going from, even though the arrows are pointing from back to the front, I'm wondering if it means that the the later layers receive information from the the first ones, because I have also.

B

Not seen it done like this, where information flows backwards.

B

Well, I if I was just visualizing what it was trying to do if I basically got a blurred, maybe edge enhanced version of the original there, and I want to basically by cycling through here, I'm crystallizing, some blob.

B

I mean I can kind of see how it could kind of reinforce itself to.

A

Center in on something, rather than drifting from side to side or something like that, but that's just how I'm visualizing, what that might look like in a graphical sense.

A

What I don't understand is how can the first layer perform, or how can the first layer receive information from the last layer when that data hasn't even passed through the last layer? Yet I don't know if you guys have anything to note or if you've seen this kind of architecture before where future information is required to compute current information, which is why I was thinking that the architecture might actually be what we usually describe as connections where information is skipped forward in the layers not backwards.

D

That's always the case for recurrent connection that you have to unroll the network to compute it, but I have never seen it like this before.

A

Yeah sort of it depends how they're doing the time. Steps too, is that like when they put that image in, did they go through and compute one iteration of every layer and then do the next one or is it just like completely pipeline? So.

B

You do you get the next image, everything just does a one-time step, and then you get the next image and you do another time steps and the information is sort of falling back. It.

A

Kind of depends how they're managing the clock, if you will I'm not sure which of those two it is, but it would seem like that. That said, even though that would be conducive to getting this backward recurrent thing working, it's very very slow like they would. I would think a an efficient implementation.

B

Matches my time you just pass in all of the frames at once. Again, I'm not trying. I would think that if, if the feedback would recall that things are temporarily displaced from the initial one, you're going to get a motion blurring effect and I'm not sure that would be something that they'll be intending, because basically you're going to see historical information, there fresh information here, they're going to be displaced and unless they're that's a they have particular processing. For that to say get motion vectors on now.

B

I would assume you pass everything through and then feed it back, so they're registered yeah and then do then do the next day, yeah yeah. That would otherwise. It makes visual hash to me.

A

All right, even if I, even if I did a single image at a time or my batch, that probably wouldn't necessarily happen right. This is, if even in a batch, I'm technically considering one image at a time. um Well, if.

B

They're all displayed you say a batch but they're displaced in time right right, but the appropriate features in a particular slice.

A

Of time are matched with the the exact same raw image. In that.

B

Same slice of time, I think they were saying that we pass it all through, but okay, if okay you're saying that you could pass a batch of them through and then they would, they would synchronize in time by when they get finished, you're, essentially looking at the elements of the batch being end of each other, nice and close I'm. This is purely conjecture on my part now I have no idea how this yeah particular module works.

B

I don't think there'd be a particular advantage in that, if you're trying, if you're trying to look at this thing as video frames, as opposed to disparate things, that are destroying the time that I think you could probably just said, but if these things actually have some temporal coherence, I think you would want to take some advantage of that.

B

So is this just for feature extraction. This is true, so this is not no graphs, no graphs here. So where is the output? Yes yeah? So that's the second.

A

Chord, this is actually the most minor part of the whole. I think, but where are the features coming out? Is it from fc6, because the arrow on the top left shows it coming out of com1?

A

This is again conjectured, but what I think is happening is that they, even though the output of fc6 gives you some output, what I'm thinking happens is they actually fetch the learn, features in conv1 and use that as the output, okay, okay and then it's congruent at the last time, step of the network being unrolled so that the output is an image of the same dimension as the input image?

A

Okay. But then what is the point? Okay, I can kind of see that I'm just then wondering what is the point of the recording, I'm not sure that's kind of weird okay, but oh, but this is trained. Is this trained in a self-supervised this? This is trained in a self-supervised way. So maybe that's what.

B

These things are there.

A

For but I even if it was trained in some supervised way, um are we treating them like a? This? Is like the decoder portion, yeah yeah. Are we taking.

B

C

By the way, I'm back and- and you guys had moved on from the same slide, so I couldn't have missed that much.

A

Yeah, it was a bit confusing slide, but I think at the end of the day, what.

B

I'll be saying is not super important. It's just a way of getting yeah.

C

I heard that so you didn't miss too much. I I I chose wisely when to leave here.

A

Okay, so that that portion is trained in this supervised manner um and they they have. This whole talk in the paper about why they can't just use a traditional encoder decoder architecture, because those latencies that you learn in that traditional architecture don't learn the physical structure of your scene explicitly um and even in the cases where they do try to learn the physical structure. The decoders are very, very highly parametrized to learn from unstructured data that you're passing at the beginning.

A

So, even though the decoder, even though the encoder is not highly parameterized, the decoder must be to learn from the unstructured data that you're passing in um so instead what they, what they do is they use something called a zero parameter decoder to enable something called painting by numbers.

A

So, given your predicted attribute vector which describes the the physical characteristics of some nodes or vertices in that layer of hierarchy, they try to paint in specific regions. Given your ground truth of your scene, segmentations and your scene.

B

Segmentations are the spirit spatial temporal illustrations that I described in the beginning, but.

A

That's standard or any.

B

Signification, that's just like a standard processor, yeah.

A

But they this is, I they don't exclusively describe how this is different from the from the first paragraph that I mentioned there or decoders are very highly parametrized.

A

um All they're saying is that there's two like they elaborate and say that there are two ways to do the second point here, um but the information that they require is only the attribute vectors, so the the physical characteristics of what you're, seeing um in that sense, it's sort of like training, a generative model to reproduce what you've seen in the past.

A

Any questions there.

A

So this is the two kind of techniques that they do. um The first is something called quadratic texture, rendering which basically impains a quadratic function of each of our texas attributes in that set of corresponding pixels.

A

um So the the quality thing that I've written there towards the end basically shows you, the mapping between that vertex and the corresponding pixels in your base, tensor or whatever tensor you're, using for that hierarchy, and this represents the raw pixels note that, for this particular format, what you do need is you need the ground truth um uh pixels, that you're looking at in order to indicate it and see how they how they differ from your predicted one to the ground truth.

A

But the second technique, apparently does not require the the spatio, the spatiotemporal registration and the second technique is called quadratic shape, rendering which basically predicts the 2d silhouette of a node in that input scene again, given the vertex's attributes um in this case the like, I said, the rendering does not need the actual ground truth um pixels, since all we need, are the the.

B

B

A

So this this section was very, very confusing and they point to a lot of different papers that have that have done this stuff before um and they do the same for describing their their loss functions for training these two uh self-supervised strategies. They say that this has been done before in this work um or various works. I did not read those works, I'm not including that here I just wanted to include this as two basic techniques that they do to enable their cell supervision.

A

um Again, I don't think this is particularly relevant to how they build the graphs, um since these techniques have been done before in terms of self-supervised painting again. The more the most critical thing here to note is this second point, which somehow means that this encoder decoder architecture that they're using somehow learns the physical structure of the input.

B

Better than traditional encoder declutter architectures, I just want to note that this is what they say. I don't know how this happens. I just want to note that it happens. So can I think of this, as you watch a video sequence moving around some objects and this this whole system is segmenting out independent objects and saying okay, this glasses, these pixels and this phone has these pixels and.

B

As you can see, in.

A

This pipeline this site, this, this module here, happens sequentially with other modules so over time. That would enable us to to keep going um further and further into detail as you explore particulars, so this thing will basically segment out the different objects and sub objects.

A

Yeah yeah seems like that's, that's almost a harder problem with building the graph itself. The the graph building is harder because you, you need to understand how specific nodes are tied to each other, um and you need to do that also in an unsupervised fashion. I'm likening.

B

A

To semantic segmentation, but done in a self-supervised way, so you have no idea what what objects you're you're segmenting! You just know that you're segmenting, somewhere, um okay,.

B

Is it any more complex than it finds out which groups of pixels seem to move together.

A

It's not more complex than that based off of how it gets intuitively. That's what's good, I'm sure the heuristics that they use are actually similar to what you just mentioned in that they use I'll, explain that in a second. um But again it seems like this is the relatively trivial part of this paper described by them. Then the further details go into how they explicitly build this graph from these segmentations um and again.

A

This is this is done in a self-supervised way and if you look at, if you look back to the top left, um that's the features that we that we get from this extraction process. It's not completely segmented. There, the portion, the we try to segment things that are similar in in structure, but the real um separation happens when you actually build the graph. So you try to understand how does one node, um how similar is one known to another?

A

That doesn't happen completely with this convoy, so this is like a building block on top of which they they built. The graph.

A

Yeah, so this is the last slide for feature extraction. They have the next set of the pipeline, um which is called graph. Pooling um and graph.

A

Cooling is probably one of the most important parts of this entire pipeline, and this effectively creates the within layer edges between nodes at a given hierarchy and using this they create the next layers of vertices, and in addition to that, they also create the child to parent edge structure that goes from one layer to another and also the registrations that describe how particular nodes that you now construct um refer back to the uh the raw data raw pixel data and the way that they do this I'll go step by step, since they create a lot of different things.

A

I'll go step by step. The first thing that they do is they create the within layered edges and they use this. They do this by giving the data from your attribute vectors at that same hierarchy and the the way they do this is they basically train something called a learnable affinity function so for any two pairs of vertices in a specific layer, the output, the probability that those two vertices are connected.

A

Given your attribute vectors for those two vertices, so the loss functions um that they describe here are based off of two two heuristics that they've been, but they say, um and they call this um something like perceptual grouping losses um and again these are heuristics based off of how physical objects are supposedly um supposed to interact with each other um in the real scene. The first one is called attribute similarity so nodes whose attributes are similar to each other um and have close space.

A

Geotemporal position should be grouped together, um so this might be like they have similar color or similar.

B

Like an edge continually, yeah could be an edge yeah, both edges of similar edges or something like that or similar textures.

A

Right and when I say position, I don't mean exclusive explosively displaced, I just mean how close are they together um relative um relative to to other objects, as you see them through all the frames, um so those nodes who have similar characteristics in color and shape and are closely positioned together, should probably be joined together.

A

That's the first heuristic. The second heuristic is something that they call statistical, co-occurrence, so nodes that appear often together throughout time throughout your image.

A

Frames should also tend to group together, so using these two heuristics, they effectively train an auto encoder to encode the pairwise differences and use reconstruction error, as an inverse measure of affinity, so affinity is, is um how what's the probability of two nodes um getting joined together in that hyper edge event that I described in the beginning, um so you can think of minimizing loss in this auto encoder as an increasing affinity.

A

How did they know how many vertices are at this level? Yeah? Exactly so that's uh I think I described that yeah. I describe that in the next slide. um Briefly, they don't um okay, it's it's, since it's done in assault supervised manner. um There is no it's not like k-means clustering, where I know exactly how many clusters I want it just keeps going. um I think, until some threshold is reached.

B

Oh okay, so they try to figure out how many words.

A

So the the vertices are implicitly described by this last point on this slide, so probabilities between um the connections between two vertices are thresholded in an adjacency matrix, so I can't technically just keep going on and on forever if my probabilities are thresholded. So let's say: if two vertices have some statistical co-occurrence that is greater than 0.5, then I decided to to threshold that and add that to my new hyper edge in that in that next letter. If not, then I don't do anything so technically the process stops once I'm under some probability threshold.

A

What, like you said, there's no fixed set of here's. How many vertices I wish to create in this layer, which is.

B

Hey, that is a benefit yeah, so after that process is done, graph pooling.

A

Also clusters the resulting graph using something called label propagation. So it's worth noting at this time that we already had this. We already had the vertices in that layer and we've predicted this we've predicted the hyper edges between the vertices and that layer. So now the second portion is something called label propagation.

A

So for every single vertex in that layer, we basically assign a random unique segment label to that vertex and perform some n number of iterations of updating each vertex with the most common label in its radius. um What's a label, so a label here is the way they've described in the picture. Here is based off of colors.

A

They just assign it some random label like a random color um and they try to find out um what's the most common label in that radius and the ones that are the most common um I think get assigned to the ones that are most common are connected to each other, as you can see in the second lecture, so the first picture assigns all the labels randomly and the second picture draws connections between um between labels that are most common in a specific radius.

A

I don't know how exactly this this process works, because the labels are random. Then.

B

There's no, then there's no disconnections yeah, I'm interpreting this like it's just like k-means clustering, you initialize the labels randomly and then you start reassigning labels in each iteration and there's a theorem that converges. This just adds the extra step instead of looking at all the labels. You look at in some epsilon ball around each node, so this includes like some type of spatial information. On top of it.

B

I don't know if that's really right. Oh that's.

A

The way I'm intuiting this okay, so like the that's something that I don't think was explicitly described here because they say that label propagation has been done so much before.

A

So just take a look at x and y work, but I think what you said just might be pretty crucial in the sense that um part of this is re-updating the um or reassigning the label based off of uh giving you random initializations, just like including this cluster, um so some label most of the labels will probably be wrong, but you just keep re-updating them over and over.

B

As you know, as your cluster, is each label unique or is, do they find some ways of having some of these nodes actually have the same label even though they're physically disparate? That's, that's this. This.

A

Question I think, um in the sense that in the beginning they are unique, but there is some re-labeling process that happens. That makes them not unique towards the end, because I think the thing where you're looking.

B

At how the attributes, similarly are various other things would be a categorization mechanism. At some point it is, it is.

A

But that's in the edge- that's in the last! That's in this slide, where, um if I want to create the hyper edge between the vertices- and I want to know how similar one node is to each other based off of their attributes, their physical characteristics. Okay, in this segment, all you're doing is clustering, the graph um explicitly so you're using the edges that you described and you wish to cluster them to form um like some clusters. That's what I'm sorry! So in the.

B

Previous slide, are we talking about these these nodes, um I'm trying to figure out what the labels uh the labels are merged out of what you used to call a node in the hypergraph or whether this thing here is we're actually labeling. Once it's been identified, are you referring to labels as labels here.

B

Yes, I'm trying to I'm trying to figure out whether the two slides are at two different levels of somatic hierarchy or whether we're okay.

A

So the previous one, all it does is for a given scene. It tries to understand how specific vertices are joined together in a specific hyper image, but what I haven't done, if I only did that step, what I haven't done is understand how one cluster relates to my parent cluster that I had before and that's what this is trying to do. I see okay, so I'm trying to set the hierarchy. So this is this is what sets the hierarchy. So the last sentence here is the most critical.

A

The resulting clusters form the next layers, vertices with child to parent edges, defined as the map from the original vertex to the new clustered label.

B

I think kevin had a good question that was also digged into that bigger question. um If we can run with the k-means analogy, k is like a parameter that you select up from. So how do they decide or like do they decide?

B

A

How do you decide okay things and the clusters that you're forming here are explicitly related to the number of hyper edges that you performed in the previous step, and the number of hyperedges that I formed in the previous step are done purely by probability threshold. So there is no limit to how many I can create until I've until I've come below with some some until I come below some probability of measurement.

A

That seems like that is telling you how many, basically, how many vertices.

B

There are in this group how many.

A

B

Are, in other words, k is a different thing if I understand it directly.

A

What I try to when I try to think of the word clusters here is how does one edge and all the vertices in that edge correspond to the next layers? Cluster or sorry, the next layer is the edge and the next player is set of vertices in that edge. That's what I'm thinking of as a cluster. I might be wrong, but that's why the clusters here don't need. uh So the last point.

B

Here I say I can create an arbitrary number of clusters, the way I'm visualizing. It is as explicitly dependent on how many um nodes that I started off with in that edge and that's what's our maturity about this.

B

It sounds like at the very, very lowest level, there's an aggregation mechanism, that's fundamentally based upon properties that you're not going to understand, but then, as you ascend the hierarchy, it becomes more and more abstract similar to this thing, so I'm just kind of curious to as you go up to the hierarchy, does it still have this notion of of attributes being propagated up, or is it purely some other abstraction from that?

B

You have these attribute vectors, which we just find very abstractly. I just I don't know if the first layer is just grounded in something the other ones are.

A

You're saying how do I preserve granular features as I keep going down.

B

Right, I'm saying you start out at the bottom right: okay and you start seeing these things start to emerge. So with that first level of aggregation, you're looking at pixels, so, okay, I kind of clustered them. I got the moving things. I got all kinds of stuff, but then the next level up do you still have those probabilities that you had in the first level.

D

A

So that's that's actually, the now that I understand your question. That's the next stage of this pipeline. That's called graph vectorization! That's when this explicit aggregation happens. Okay,.

A

Sorry again, this might be very confusing this this this one last one um yeah I'm getting a bit lost into where this falls into the whole thing. I don't know if there's like an end picture of what their graph looks like or there is I can I I didn't want to show that picture in the beginning, because it's a huge pipeline, it's incredibly confusing, so I'm trying to take in peace.

C

I think I think what we're doing is we're spending a lot of time on the pipeline. Maybe super tight suggesting would be helpful to know where we're ending up, um because you know we have to ask ourselves well how many of these details really matter to us or not right.

A

This is what oops this is what it looks like at the end, um if I had if I started off at that very beginning, and I go through this this pipeline- that's this is effectively what I what I have at the end, the the middle picture or the sorry.

A

The second picture takes a look at takes a look at the code that I have, I think um in the original raw data, and I keep segmenting that over and over until I get some components in that code- and this is the graph this is, these are the hyper edges of the first level. These are the hyper edges of the second level, and these are, I guess, all the words containing that single hyper edge that we're working on.

A

Okay, so the middle one might be the christmas tree thing yeah. I I think these are all segmenting. The christmas tree like it goes deeper into the christmas tree.

C

Like I'm sorry, I'm confused by that you're, saying you're, saying that the third image here is just the christmas tree part or is it is how the.

A

C

A

I, I think, that's, I think, that's a better way of looking at this. Yes, these are all clusters within that christmas tree.

C

And then the final, what's the final image then represents the final.

A

One is entirely abstract: it could be any one of these, but it's some part of that christmas tree that the the system has potentially recognized as a sub-object. Within this, this cone.

C

I would thought that the one on the right would be the christmas tree relative to other things in the scene. Isn't that the overall goal here is to figure out the scene graph.

A

A

B

A

At it is, if I started from all the way at the end and worked my way to the top.

C

Which what do you mean the left side? Point.

A

To the right, the right side, all the way to the left side, then I'm starting with the most granular features some aspects of the cone and I'm working. My way left, left and left.

C

Until I get to the cone, but but isn't the goal of this to do a scene graph, this.

A

Is like this is a scene graph in the sense that I've I've, segmented.

C

Various points of my scene, I understand, but just showing that knowing where the tree is, is not the scene graph right, the only. I I'm sorry basically confused. I thought in the end, you would end up with like here's, a graph representing all the objects in this scene where they are relative to each other.

A

Oh with, like with labels, you need.

C

Or just just a graph representing the scene, whether they're, labeled or not, but like. If I'm hearing what you're saying what what this picture here represents is just figuring out that there's a tree.

A

The this is probably the wrong picture to show them. I I know exactly what you're asking for, and I can show that after the presentation it's in the.

C

A

C

That's that isn't that the goal of the paper to do scene, analysis.

A

Yeah so that particular that image I left out, because it was like a weird 3d rendering of this entire graph. But I I think I understand what we were asking.

C

For so, this is a subset of the whole thing this image here represents. How do we figure out? There's a tree, and the tree itself is a graph of different levels of hierarchical composition,.

A

Yes, yes, and you can use this information to build a 3d render of your scene where each individual part of that scene represent is represented by um or you understand how each object in that scene is in relation with the other objects in that scene and yeah.

C

Like this, this picture doesn't show that right, yeah, okay, this just picture shows how the different parts of a tree are related to other parts of the tree. Yes- and this is.

A

I mean this picture also doesn't show the full picture in the sense that I can't, if I wanted to show you um if I zoomed in on this hyper edge here, I could- and I could show you all the different components of the ground on which the ball and the cone lie.

A

This is only this is just for clarity that it's only I'm pretty sure it's only zooming into the code, but this is like a multi-layer, um expensive graph, that's growing, so it would expand into all these individual hyper edges that you started off in this player.

A

Hopefully that gives some idea of the end goal. I can show a better picture towards the end. um I can go back to maybe the.

C

Oh well, I guess the end goal would be a scene graph but but you're saying maybe you know.

A

Yeah, maybe I can show that explicitly after after this there's only one or two slides here, and this describes the aggregation process. um So this module here is called graph vectorization and it happens in sequence, with graph pulling and it happens over and over again- and this is how you segment for them for the image that you have so this effectively aggregates the initial attributes that you started off with, according to the edge structure that you predicted from graphport.

A

So the aggregation step is actually pretty important because it preserves the the details of this graph as it becomes coarser and coarser and higher levels, while still allowing you to predict new attributes to encode scene properties that aren't just simple functions of your low, lower level nodes that you started off with. So what does this mean? This effectively means that I don't have one of two things happening. One is as I go deeper and deeper into a particular object.

A

I don't just lose information about where it was with respect to the original nodes that represented the whole object. That's one, and the second one is that those lower level properties. uh Sorry, those those course properties are not just functions of the original objects um or the nodes of the objects that you saw in the beginning, so it works both ways. um This aggregation step is pretty confusing. um They they they kind of uh jump over this.

A

They say that they tried a few different methods to to to do this aggregation step and they show us two of the methods and I'll show that in the next slide, but um assuming that this aggregation is done using some kind of techniques that they've listed, they then use some regular multi-layer perceptrons to approximate something that they call graph convolution, which allows information to be exchanged between vertices in this graph.

A

And finally, the output of this graph factorization is the output of this mlb, and this transforms your your your initial attributes, um a of l into the next layer's attributes, a of l plus one. So what what is the goal of graph vectorization it's to preserving yeah? So it's it's to.

A

If I the way I saw this was, if I didn't have graph vectorization, then I wouldn't know how one node communicates with another node. I have no way of of I've. I've I've talked about that I've. I have all this information about each individual node, but I have no way to understand how they work together. um If I wanted to have, if I had several nodes to represent, let's say this this class here, I have no way of aggregating them into one node.

A

It describes all the properties of various nodes that represent this, and this is necessary for me to build on to the next letter. I need to start off with a single vector that represents everything that I've found. Is it just clustering.

B

Vectors into what might belong to the same object at that level, it's not quite that.

A

Close because that was the example- that's already happened. This is this is trying to um okay. If I had individual vectors that described potentially different parts of this of this of this bug, all the aggregation step is doing is joining all these individual vectors into a new vector that describes one particular aspect of this of of this class and that particular vector is used on the next layer to build even further information.

B

About, I guess another part of this, so it's kind of summarizing yeah in some sense summarizing it into a new representation. Correct. That's um that's capturing multiple vertices, yes, okay! So just some additional thinking here this this looks exactly like crafty learning and you can say: they're approximating graph convolution, it's more complicated because you're building the graph like you're deciding what are the nodes and the edges as you go so you're like building the graph and then doing graph learning on it simultaneously, but like one of the.

A

B

That you would do with gravity learning is just trying to get a vector representation of a graph. That's useful for a particular thing like classifying whether or not a molecule will behave in a certain way but um like what's the loss function like what are they actually optimizing for when.

A

They try and get this vector representation of a graph yeah. So that's that's. That's very briefly. The techniques that I'll show the next slide. I have no idea how they work. All I know is that they consider two different things or a few different things, um but to to answer your original point, the only difference between this and traditional graph-based. Deep learning is that graph-based, deep learning has no need to preserve all the features that I've seen, because it doesn't need to cluster a node into a new one right.

A

I already have the graph representation, I'm just trying to understand how the data fits best with all my parentheses and nodes in that graph right, I'm not summarizing all the information in order to create new subnets or child words. That's what this step is for right. Okay, so you have like these different parts of the image. Let's say that.

B

Are segmented and there are nodes associated with each of those chunks yeah, and so this is basically doing something like graph deep learning on each of those separate graphs and each of those graphs are really part of the whole big thing. So each graph and note are sort of like interchangeable when you think about a.

B

If you just change a level of the higher a graph becomes a note.

A

Yeah- and I I get one yeah- is that the point- or maybe that is that's sort of the point, and I understand why this is confusing, because it seems like we're re-clustering everything when I already had the clusters, but that's not what this is doing. In order for me to create new information at the next layer of the hierarchy, um I need to have a summary of everything: yeah. It's like a summarization or some.

B

More compressed representation- and I can't do that if I just concatenate all my vectors in the past.

A

Because that's that's that's this point here. um I want to retain information from my base tensor, and I also want to retain information from the the features that I've just learned. The granular features that I've just learned.

B

That's this is the best way to do that. You're going to ask is: does this thing allow in principle something like I've got a bunch of n segments and somehow I'm able to group them and then deduce that they're round or something that's a higher order than just you know? Oh I've got a segment in space that is adjacent to an extra that's a closed curve, but am I able to get something a higher level of abstraction like I can say something about it?

B

I introduce that it has a property which we would fall around or something like that. That's what happens inside the attribute vector yeah! That's that's something that I've stored as I sequentially progress this graph.

B

Just because they're saying you know, there's like a canonical set of operations that they do on those things which would be any kind of feature: detection thing where you're trying to look at okay, this local region, what's first over segmental, voltage and stuff like that, but what I'm wondering is you know when you look at the child nodes, pull some kind of statistic out of them: that's either common or it expresses them as a cluster.

B

You get a boost in abstraction, somehow that you see something higher order about the thing or is it just simple atomic processes that say: okay? Well, I've got you know, n different values. For this thing I just do variants on it.

B

But this one, you know, because there was a reference there of, of getting more information, and I was wondering if this is you know, for the low-level properties of high-level properties is something like that kind of abstraction climbing something that this thing actually is capable of doing or is. Is that you you've got these ten statistical operations? That's it just looking at an affinity for those things.

A

I'm trying to understand your questions, I'm trying to think of what the.

B

Expressive power is when you move uh when you look at these child nodes and find a a you concoct, a attribute vector that summarizes those nodes, as opposed to not doing that. You're saying, as as as opposed to.

B

Something and work something interesting uh emerge from that, or is it in in that clustering process? In other words, the example I gave was you know you could say: okay, I've got a bunch of mind segments and I can say: okay, the first order. You know the moment of inertia around these line segments is something I can compute that oh okay, fine, that's it. I haven't really discovered anything or is there something that said? Oh okay, I've got another one over here. Do I start seeing something where something emerges from from that?

B

That's a higher order thing because of these techniques, or is it I only? I can just say what the first order moment is the next thing up there. The next thing up there is. Are you bounded by the statistical measures you have for these, for these attributes is what I'm saying.

A

I think this is two separate questions I don't know which which aspect you're you're focusing on the summarization just ensures that I don't miss out on features that I had previously, but this the I I don't know if you're focusing on that or if you're asking about how do I.

A

How do I understand if two nodes share similar properties? Why not, even before this step, doing the pooling stuff? um How do I know if, like you, you gave up you, give the example of of two nodes potentially being inside a round object right? How do I know that they're similar to each other? I don't know if you're focusing on the graph pooling step which effectively clusters based off of similar information or this step, which tries to summarize all the nodes in that cluster or.

A

And you're asking what properties does this allow, as opposed to me not doing this.

B

Like this is something interesting arising from this step, what I'm saying is: what's what's happening, it says vectorization well achieves by taking means variances first order moments higher order statistics over each of the vertices is that it I mean there's operations, that's.

A

It that's the only thing they look at there's more there's there's they. They experiment with a variety of different summarization factors, but whatever they.

B

Do it's consistent across the whole thing.

A

Oh yeah yeah, it's not different per person.

B

Okay, so so I guess I guess my question is: is it sufficiently rich that something like some property like roundness could be worse? Yes,.

A

Yes, I think so, okay, so um this is a very simple picture to try to understand, what's happening in that step. um If I have individual nodes inside my cluster or my my hyper edge here, I'm trying to summarize them with various different things: the first one computes the volume inside the inside a specific object and the second one computes the I think- that's the line integral to compute the contour of that image.

A

um But this is just stuff that you, you try to aggregate as much information as possible and then add all this information to your nlp to try to come up with some summary of all the different attributes inside that object. You use this summary to then produce the next layer set of of vertices.

A

Again, I don't know the specific loss functions they're using to how they train this graph vectorization module. All I know is that they consider these different statistical measures to help summarize um your your attributes inside this, this one hyper edge.

A

Okay, um the whole process. I've already shown this on see if.

B

B

A

Something that's mlp like how's that mlp learned I. I have no idea how that sounds. I don't know the loss function for that. Okay, but.

B

I mean, is it, is it? Is it? Is it trading on the fly, or is it a pre-workout? No.

A

It's training, yeah, okay. uh I had one more slide here to discuss caveats of this and instead I think I want to show the.

A

I want to show the image that jeff asked for and I can find it.

B

A

Okay, this is sort of what the end goal looks like I don't know if I can zoom in even more, I don't think I can.

A

This is what I showed you right now this this thing right here, where I'm effectively just zooming into one particular object of from the graph that I've built of that object. I keep building that graph until I get to some fine grained details about some specific part of that object.

A

I'm going to skip over this right now- um and this is just I I'm pretty sure this deals with um the label propagation and how I've, uh how I've assigned a specific node in the graph I've built to the original 2d object or to the original 2d pixels that I'm looking at. But this is, I think, more interesting, which is the 3d rendering of the graph that I've built.

A

So I start off with all these nodes in this scene in some of the nodes, I basically cluster um using the graph vectorization to basically summarize everything in that cluster that I've built through these stages and effectively after that I can label here's where the ceiling is. Here's where the wall is and here's where the code is um this is done impersonally. I would say um that's a human going in later and saying that's a four.

A

I couldn't find a video of this working, but they do try on uh the few uh environments that I showed in the beginning and there's some kind of. I can't understand what the the metric is for understanding uh the semantic segmentation they're trying to solve, but they do compare it against a few baselines and they say that supposedly I can. I can just zoom into the results here. I don't know if you guys can make more sense of this might be testing like well. What's that picture up there.

B

A

Are different scenes right.

A

Yeah, this is more like the the 2d registration things um that I was talking about. It's like highlighting the the colors based off my graph onto the original image frame.

A

This is like the explicit semantic segmentation that's happening, although I don't think this is that useful to see, because this is no different than any standard semantic segmentation tactic. What's more interesting is how the graph is generated. I know that here exists something different from something else. That's that's very close by in the graph, but is there an.

B

End goal to be able to do really good segmentation.

A

Their end goal is to have a better um is to create a yes, a segmentation tool that has a better inductive bias than or sorry a better inductive prior than cnns, which is true from the get-go right, because cnns do not do this yeah. Is that really the goal? I.

B

Would have thought the goal would have been more just like scene understanding, it's like kind of one step above higher picture since.

A

It's done in a self-supervised manner. I don't know what the individuals are here like that's why the human goes back and labels. Here's where the code is, here's where the tree is where the graph is well in some sense in 3d. If you know.

B

Where everything is, if you can segment everything out, but you sort of know the.

A

Relationships but you don't know that it's a christmas tree or it's a wild, absolutely there's some object here that belongs to these. This 3d.

B

Subset those christmas trees, this 3d exactly the volume as well, so you kind of partition them out somehow that and then, if you know this is an object, that's an object, that's not out there! You come along with attributes too. So if you've had another scene, you've got a christmas tree in it. You might be able to say something about that. Maybe I don't know if they do classification as such, I don't know if, like all christmas, trees would end up being more similar to each other than other things that are about christmas.

A

Does that sort of make sense, in the sense that the final step to get to scene understanding is having the labels of what I'm trying to predict? This doesn't do that this is more scene decomposition into a graph, which is, I think, pretty interesting. um The steps that they take.

B

So let me let me rephrase that question the reason I'm interested in this is because it seems like a step forward to see an understanding like at the end of the day. The reason you.

A

Would decompose scenes because you want to understand, what's going on automatically?

A

Is there? Is there something I'm missing there like? Is there? Is there any other reason to do.

B

This whole line of work.

A

That's correct, um john. If I didn't, I just took a traditional cnn that does scene segmentation.

A

It's uh it's going to have a very hard time understanding the actual physical attributes of individual objects that I'm looking at the only way it knows to segment is because it's seen so many different varieties of that same object during training and tries to understand their contours.

A

It tries to understand their shapes, but I think in uh at least in the uh luis was showing this in the habitat environment, where, if I just change the colors of all the the objects in my scene and whatever agent that I'm training to recognize those objects completely fails right. This one performs a little bit differently.

A

It's not doing scene segmentation just with cnn's, it's trying to understand the actual objects and then the contours of the structure of the objects itself yeah, but so going back to what you're saying ben, I think I think there's a lot more. You could do with this than.

B

Understanding uh see, the understanding would be just like the start in some sense like there's know where the objects are and where their relationships are to each other, but we could use that as a basis for learning about objects like about christmas trees and so in the future. You'll recognize more christmas trees different, so you could use.

B

I think this is more just as a starting point for everything else you might want to do.

B

So this is kind of a necessary prerequisite before you can, if you're, if an agent is.

A

Only getting visual scenes not isolated objects. You have to do.

B

Something like that in order to even get training data for something within coffee cups, and things like that. So it's kind of like a prerequisite.

B

The questions I'm asking myself internally are like what is like the overall goal, long term for this type of research. Was this a useful step in that direction and you know which techniques that they use? Can I extract right and understand and repurpose there were a lot of different techniques used in this paper. Yeah. It's like a lot of different things. It seems like I mean once you have a principle segmentation that is higher, so you can see the point wise. You can start looking.

B

You know, okay, I can track this thing in the video here and here and here. Okay, so that's the start, so I could recognize the thing. Can I use whatever that word representation? Was that attribute vector, whatnot and just blot it on? I said another random see something.

B

Can I lift something out of a completely different scene? That kind of tells me that there's some similarity there.

B

Is you know it's done on? You know one sequence here.

C

What do you give a fresh sequence? Is it? Is there anything retained, differentiated or does it start off from scratch again.

C

They tried that this is my question. They don't they don't, but I'll be. Are we are you? Have you presented the everything you wanted to present.

A

Oh yeah, I just I was just going to show the results next, but.

C

Can we go on to that sure.

A

C

A

Your question towards the end: okay, okay, um that that picture before that I was showing kind of tells you that they're trying to use the graph that they've built to solve the same scene, segmentation tasks that other cnns are trying to do, um and I think the metrics that they do that. Try to do that with are like uh bounding box overlap, standard metrics, um it seems it seems, like risk, performs better than some of the baselines that they compared against. um But that's that's not. The point I wanted to highlight.

A

The point I wanted to highlight was there's a section in this paper where they say that um the generalization for this particular task is significantly better than than any of the other baselines that they compare against, presumably because those baselines are not building their own uh graph representations, whereas this does um and that's kind of where I wanted to get into the the caveats and important takeaways of this paper and I'll share. The last slide that I have to show.

A

So the premise is to present an approach to segment scenes or to come up with some way of of uh of separating objects in scenes, with better inductive bias than cnns, and, like I said in the beginning, it should really come as no surprise that this is better because the cnn cannot break down the scene into individual components, there's a lot of training tricks and details to enable self-supervision that I've glossed over a lot um halfway, because I don't understand it completely and the other half is because I don't think it's it's it's important to understand the pipeline itself.

A

um The pipeline itself is tailored to pick the best visual features in the network, um and this is not just in the the feature extraction part. This is also in how the aggregation step is working. um This is all manually done, so the graph can effectively preserve important features, as the graph is getting built.

A

The key thing to note here is that there's no learning of pose of orientation in the graph, it's purely how one object is physically related or located next to another object.

A

The initial results here show that it can seemingly generalize generalize better than baselines, but, of course this is, uh this is building an explicit graph. Other baselines might not, and the test environments are pretty simplistic, there's nothing changing in scenes. The only thing that's changing is my camera angle, with respect to some static objects in that scene,.