The spatially-embedded
Recurrent Neural Network

Project overview 

Summary 

• RNNs faced with task control, structural costs and communication constraints configure themselves to exhibit brain-like structural and functional properties.

• These spatially-embedded RNNs show:

  1. 1. A sparse modular small-world connectome

     2. Spatial organization of their units according to their function

     3. An energy-efficient mixed selective code

     4. Convergence in parameter space 

Background & question 

• Due to being exposed to the same basic forces and optimization problems, brains commonly converge on similar features in their structural topology and function [1].

• Can we observe this process in a recurrent neural network’s (RNN) optimization process of task control (one-choice inference task) under structural cost and communication constraints?

Our approach 

• Spatially-embedded recurrent neural networks (seRNNs) are characterized by a special regularization function which embeds them in a 3D box space with local communication constraints.

• We trained a population of 1000 seRNNs (10 epochs) and compared them to 1000 L1-regularised RNNs (baseline models). In both populations we varied the regularization strength systematically.

Structural findings 

• As in empirical brain networks, seRNNs configured themselves to exhibit a sparse modular small-world topology [2].

Structure-function findings 

• Mirroring neural tuning [3], functionally similar seRNN units clustered in space. Like the brain, task-related information has an organized spatial configuration. 

Functional findings

• seRNNs exhibit a mixed selective [4] and low-energy demand code for solving the task.

Convergent outcomes 

• Findings emerge in unison in subgroup of seRNNs within a ”sweet spot” [5] in the regularization strength and training duration parameter space.