DevoWorm Research Presentations, 1 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Neuromorphogenetic Patterns and the Theory of Deep Learning

Description

Contributors: Neuromorphogenetic Patterns and the Theory of Deep Learning. Poster presented at NetNeuro 2021 (Satellite session of
Networks 2021).

Contributors: Bradly Alicea, Mayukh Deb, Mainak Deb, Krishna Katyal, and Jesse Parent

A

Hello, my name is bradley alicea and I'm presenting this poster neuromorph genetic patterns and the theory of deep learning and my co-authors are mayukdev mynoc, dev, krishna, katyal and jesse parent, we're from the diva worm group, which is part of the open room foundation, and uh this poster is about the theory of deep learning it's about deep learning networks, but also how they apply or how they maybe can be informed by biocomplexity and by biocomplexity we mean brain networks, but also developmental self-organization, and so why are? Why are these sorts of themes important?

A

Well, as you can see, we have a difference between brain networks and deep neural networks in terms of topology in terms of structure in terms of output, but we can also inform sort of this gap through looking at how sort of developmental structures or self-organized structures are formed and the sorts of things that they contribute. So it basically provides an extra dimension to this deep neural network topology, and so the outcome might not look like a brain network. It might look like a developing organism. It might look like a developing brain network.

A

We're just exploring this theme in this poster, uh so we're basing our sort of assumptions on this idea of computational equivalence, so computational equivalence is that the systems found in the natural world can be perform, computations up to a maximal or universal level of computational power. Basically, we can take a natural system.

A

We can abstract away a lot of its complexity and we can end up with this fundamental sort of skeleton which you can see in the deep neural network, and if we get that exactly right, it will be equivalent or be very similar in a lot of ways to your biological system. That's our the idea of computational equivalence. We can argue about. You know whether that's actually the case, but we have to find the right sort of representation for that, and this is why we're introducing a developmental aspect to this as well.

A

So in this part of the poster we talk about. This is an introduction to this idea. We kind of focus on gans, so we focus on a certain type of deep neural network.

A

Gans or generative adversarial networks are networks that have two components: they have a predictor component which takes data or information from the environment.

A

It classifies it, but then there's also a an adversarial component that generates fake or adversarial data, and this adversarial data is then contrasted against the real data and it helps or maybe even hinders, the learning of a network. So the network can learn things. They learn things not only from the environment, but they learn from the adversarial set, which is generating a bunch of fakes, and so you can see in this figure.

A

This is a generic figure of a gan where you have a fake image and you have a training set of real images and it's training it to discriminate between the fake and the real image and and if it's a successful uh model, it will be able to successfully discriminate between the fake images and the real images in the training set.

A

So we have a couple of uh sort of topical areas that we want to focus on. For this, we have first, this feature discovery. So why is this approach that we're proposing?

A

Why would it be superior, or why would this sort of idea be a superior direction to go in? uh The first is feature discovery, so in gans we know that feature discovery, proceeds in part through an adversarial approach.

A

uh Pens not only offer novel mechanisms such as adversarial training, but also shows that game theoretic approaches can be applied to understanding networks that control morphogenesis and development. So there's this aspect of morphogenesis and development, and here we have a sort of a preview of some work that our group has been doing on what we call morphogenetic agents which combine pattern, production which is morphogenesis, in other words, these agents that produce patterns and present them to an environment.

A

Where, then, you have these other agents that pattern recognize or try to recognize these patterns, and and then you know they can recognize either the other agents as prey where they can recognize and distinguish between different types of patterns that are generated in the environment.

A

So these this is a sort of a co-evolutionary training mechanism that might be something in a direction to go in. Another direction is that neural networks provide us with both circuits. These are network subgraphs that serve as a set of solutions and universality, so features and circuits that generalize across different systems and contexts.

A

So you know we want to view neural networks, maybe as sort of a universal type of nervous system, and so we can look at different organisms and different uh types of instantiation of nervous system and compare that with deep neural networks or different types of designs of deep neural networks, and this will yield some interesting information. I think about both artificial systems and the natural systems.

A

The second is network depth, so the gans framework utilizes deep learning, uh deep neural networks, of course, as their core, and so unlike cellular automata, which are these structures here which operate as a lattice, where the neighboring cells all influence, each other's state and standard neural networks can exploit many layers of processing specifying the generation and recognition of classes of specific features within a complex pattern.

A

So we can actually classify things in the environment using these, but we can also link them to neural networks and the way neural networks do can do the same thing. So we've done some work in our group with neural networks trying to produce embryonic shapes, but also classify embryonic shapes, and so we can do the same thing with gans to an extent.

A

We also can link this to the production of phenotypes, so we also have a image here where we talk a little bit about the formation of phenotypes.

A

The second part here is that, well, there are two parts here, so the first part is high layer. High numbers of layers in a biological network provides a conduit between the genotype and the phenotype, so that mediates complexity in various ways. Other analogies of modern, deep learning networks. So again, we know in, for example, in a biological organism. There are many layers between the genotype, the production of product gene products and the phenotype, which is the expression of gene products, but also a product of interacting with the environment.

A

Can we find analogies with deep learning networks? So all these layers in a deep learning network are somewhat generic, but maybe we can, you know, have a sort of an orthogonal aspect to it, where there's like a genotype that maps to a phenotype that then maps to these layers of neural network.

A

The second point here is that depth of a deep learning network allows for so-called cheap learning or other types of learning strategies that have been discovered in deep learning networks and there's some references here where I cite one or two of these papers, and so this is an example of cheap learning, for example, or computational networks operating within physical constraints, which can be analogized to mechanisms of facilitated variation in biological adaptation more generally, so this figure in particular, is a figure of uh facilitated variation and how it works.

A

And if you want to read more about that concept, you can go to this citation, but yes, we're looking at how to analogize learning in deep neural networks, even maybe even learning in brain networks, with facilitated variation in biological adaptation.

A

The final part of this is network heterogeneity, so future dl models may allow for network heterogeneity, which has been defined identified as a factor in making artificial networks more like intelligent biological systems. So again, these networks are generic. Brain networks are not generic brain networks have a heterogeneous structure.

A

Deep neural networks are more uniform now we're, assuming that this heterogeneity is really the secret sauce that makes brain networks work and do a lot of different things, and sometimes it's because you have multiple functions that are sort of in the same network in the same space, but nevertheless that heterogeneity needs to be captured and replicated.

A

So this is the thing that we're trying to focus on in this section and the the idea is basically a generic call for artificial network systems like this to be heterogeneous.

A

So how am I deep learning uh how my deep, uh deep learning networks incorporated into brain networks that exist exhibit skill-free distributed rich club networks, so you could have something that is, uh has multiple aspects of say: brain networks, that couple pattern generation with embodiment perception, so that goes back to these morphogenetic agents. But more generally, we want to build agents that have a body that have this sort of four-dimensional structure.

A

With this three-dimensional structure plus time, we want them to behave in the real world, and so to do that, we need to give these networks some sort of structure that resembles the brain, but also structure that allows them to process information in multiple scales, time scales and spatial scales. And what have you future? Artificial neural networks need to map to distributed decentralized topologies by mimicking general system functions such as opponent processes.

A

So we can also look towards things that we can observe in the brain, such as opponent, processes or other types of uh general behaviors, both within the network itself, within the brain and behaviorally outward into the world and opponent processes, of course work in the brain- and this is maybe a first step in terms of like replicating uh output behaviors.

A

So this is a systems sort of a regulatory process, but also something that relates to behavior. If you go to the behavioral literature, there are a number of different areas in an emotion and in other types of behavior, a behavioral approaches where this is a relevant thing.

A

Finally, the future also involves graph neural networks, which can be structured in ways similar to complex networks, so graph neural networks are neural networks that are organized on the principles of graph theory, the principles of complex networks, and then the question is: is that this? Can this approach allow us to simulate brain networks to understand the functional role of certain motifs and structural configurations?

A

So here are some generic diagrams of graph neural networks, and this is just again to wonder what the direction forward is. So these are the references and in the reference we have a preprint from our group number 16, which lays out a lot of some of these topics in more detail. Thank you.