FINDING PREDICTABLE LOCAL CLASS REPRESENTATIONS

This experiment was motivated by example 1 (see section 1).

At a given time, the `next' symbol emitted by a very simple stochastic `language generator' was not precisely predictable from the `previous' symbol but belonged to a certain class defined by the previous symbol. During training, at a given time $T_1$ saw the previous symbol while $T_2$ saw the next symbol.

$T_1$ minimized (3), $T_2$ minimized (2) with $D_2$ defined according to equations (5) and (6). 10 test runs with 15,000 training iterations were conducted. $T_2$ always learned to emit different localized representations in response to members of predictable classes, while superfluous output units remained switched off.