EXAMPLE 3: Discovering Factorial Codes

In this section, we go back to the general definition of redundancy in the introduction. There it was mentioned that the ideal of a low-redundancy code is a factorial code (e.g. [1]). Recall that with a factorial code, the pattern components of the code vectors $y^p$ (representing input patterns $x^p$) are statistically independent:

$$\forall p: ~~ P(x^p) = P(y^p) = \prod_i P(y_i = y^p_i).$$ (1)

The next section presents a predictor-based method designed to find binary factorial codes. With binary factorial codes, equation (1) implies

$$E(y_i \mid \{y_k, k \neq i \}) = E(y_i)$$

for all $i$ (here $E$ denotes the expectation operator).