RNN main page
Evolution main page
Jürgen Schmidhuber's page on
wheel of the real Robertino robot
3-wheeled robot has learnt to balance two poles on top of each other

COEVOLVING RECURRENT NEURONS

RNNs control fast weight nets for robot control
above: wheel of the real bot
3-wheeled reinforcement learning robot (with distance sensors) learns without a teacher to balance two poles with a joint indefinitely. The neurons of its recurrent neural networks (RNNs) co-evolve.
jointed pole about to crash
Above: 3 RNNs compute quickly changing weight values for 3 fast weight networks steering the 3 wheels of the robot living in a realistic 3D physics simulation. Left: still trying to learn to balance the two poles.
More about ESP in the page of Tino Gomez

More on Robot Learning

Paper: F. J. Gomez and J. Schmidhuber. Evolving modular fast-weight networks for control. In W. Duch et al. (Eds.): Proc. ICANN'05, LNCS 3697, pp. 383-389, Springer, 2005. PDF.

LEARNING TO CONTROL FAST WEIGHTS

 Cogbotlab .
cart with long pole and short pole
More work on coevolving recurrent neurons:

F. Gomez and J. Schmidhuber. Co-evolving recurrent neurons learn deep memory POMDPs. In Proc. of the 2005 conference on genetic and evolutionary computation (GECCO), Washington, D. C., pp. 1795-1802, ACM Press, New York, NY, USA, 2005. Nominated for Best Paper in Coevolution. PDF. Simultaneously evolves networks at two levels of granularity: full networks and neurons. Applied to POMDP learning tasks that require to create short-term memories of up to thousands of time steps, the method is faster and simpler than the previous best conventional reinforcement learning systems.

Related work on fast weights: J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural Computation, 4(1):131-139, 1992. PDF. HTML. Compare pictures (German).
A slowly changing, gradient-based feedforward neural net learns to quickly manipulate short-term memory in fast synapses of another net.

More fast weights: J.  Schmidhuber. Reducing the ratio between learning complexity and number of time-varying variables in fully recurrent nets. In Proc. ICANN'93, Amsterdam, pages 460-463. Springer, 1993. PDF. HTML.
In a certain sense, short-term memory in fast synapses can be more efficient than short-term memory in recurrent connections.

A related co-evolution method called COSYNE:

F. Gomez, J. Schmidhuber, R. Miikkulainen. Accelerated Neural Evolution through Cooperatively Coevolved Synapses. Journal of Machine Learning Research (JMLR), 9:937-965, 2008. PDF.

F. Gomez, J. Schmidhuber, and R. Miikkulainen (2006). Efficient Non-Linear Control through Neuroevolution. Proceedings of the European Conference on Machine Learning (ECML-06, Berlin). PDF.
A new, general method that outperforms many others on difficult control tasks.

Related work on evolution for supervised sequence learning: a new class of learning algorithms for supervised RNNs, which outperforms previous methods: Evolino (2005).

Fibonacci web design
by J. Schmidhuber