Limitations and Extensions

When both $F$ and $S$ are feedforward networks, the technique proposed above is limited to only certain types of time-varying behavior. With $\sigma$ being a sum-and-squash function, the only kind of interesting time-varying output that can be produced is in response to variations in the input; in particular, autonomous dynamical behavior like oscillations (e.g. [Williams and Zipser, 1989]) cannot be performed while the input is held fixed.

It is straight-forward to extend the system above to the case where both $S$ and $F$ are recurrent. In the experiment below $S$ and $F$ are non-recurrent, mainly to demonstrate that even a feed-forward system employing the principles above can solve certain tasks that only recurrent nets were supposed to solve.

The method can be accelerated by a procedure analogous to the one presented in [Schmidhuber, 1991b].

Back to Recurrent Neural Networks page