REDUNDANCY REDUCTION FOR SEQUENCES

This section shows how to ``predict away'' redundant information in sequences constructed from a finite set of possible input symbols. We pre-process input sequences by a network that tries to predict the next input, given previous inputs. The input vector corresponding to time step $t$ of sequence $p$ is denoted by $x^p(t)$. The networks real-valued output vector is denoted by $y^p(t)$. Among the possible input vectors there is one with minimal Euclidean distance to $y^p(t)$. This one is denoted by $z^p(t)$. $z^p(t)$ is interpreted as the deterministic vector-valued prediction of $x^p(t+1)$.

It is important to observe that all information about the input vector $x^p(t_k)$ (at time $t_k$) is conveyed by the following data: the time $t_k$, a description of the predictor and its initial state, and the set

$$\{ (t_s, x^p(t_s)) ~~with~~ 0 < t_s \leq t_k, z^p(t_s - 1) \neq x^p(t_s) \}.$$

In other words, we can forget about the predictable input vectors. We need to look at the unpredictable inputs only. Only the unexpected deserves attention [21]. We apply this insight to sequences generated by the modified automaton above.