Peter Tino
(joint work with Barbara Hammer and  Alessio Micheli)


Recurrent/recursive networks as non-autonomous dynamical systems 
- lessons learnt

A large body of work has been done on the connection between
recurrent neural networks and dynamical systems. 
For example, we were able 
to better understand why it is so difficult to learn to latch 
an important information that appeared in the past, 
and consequently construct models that are less prone to 
the "curse of long time spans".
Basic tools of dynamical systems such as linearization of the dynamics
around fixed points and bifurcation analysis enabled us to study 
learning and representational issues in recurrent networks. We have better
understood the impact of various forms of dynamic noise 
on the computational power of recurrent networks.

There are several lessons to be learnt from realizing that
prior to training, recurrent nets 
are often initialized as contractive (Lipschitz continuous)
systems that readily represent Markov models.
This has direct consequences for the style of
reporting of experimental results. For example,
in certain circumstances variable memory length Markov models
should be employed as the null hypothesis against which
recurrent networks should be tested.
Also, a detailed theoretical learnability analysis 
in the PAC framework, as well as fractal analysis 
of the dynamic activations patterns is possible.

Similar analysis can be performed on recursive networks
capable of processing more complex data structures such as trees.
I will report some preliminary results showing that 
- the Markovian architectural bias still holds
and 
- detailed fixed point analysis of the dynamic maps
  is very helpful for understanding typical patterns
  of internal representations of structured data.


What new lessons are there to be learnt from viewing
recurrent/recursive networks as dynamical systems?