LOCAL CONDITIONED VARIANCE MAXIMIZATION

This is the author's preferred method for implementing the principle of predictability minimization. It does not suffer from the parameter tuning problems involved with the $V$-term above. It is extremely straight forward and reveals a striking symmetry between opposing forces.

Let us define

$$V_C = \frac{1}{2} \sum_i \sum_p (P^p_i - y^p_i)^2.$$ (5)

Recall that $P^p_i$ is supposed to be equal to $E(y_i \mid \{y^p_k, k \neq i \})$, and note that (5) is formally equivalent to the sum of the objective functions $E_{P_i}$ of the predictors (equation (1)).

Like in section 4.6 we drop the global invertibility term and redefine the total objective function $T$ to be maximized by the representational modules as

$$T = V_C - \gamma H.$$ (6)

Conjecture. I conjecture that if there exists a quasi-binary factorial code for a given pattern ensemble, then among all possible (real-valued or binary) codes $T$ is maximized with a quasi-binary factorial code, even if $\gamma = 0$.

If this conjecture is true, then we may forget about the $H$-term in (9) and simply write $T = V_C$. In this case, all representational units simply try to maximize the same function that the predictors try to minimize, namely, $V_C$. In other words, this generates a symmetry between two forces that fight each other - one trying to predict, the other one trying to escape the predictions.

The conjecture remains unproven for the general case. The long version of this paper, however, mathematically justifies the conjecture for certain special cases and provides some intuitive justification for the general case (Schmidhuber, 1991). In addition, algorithms based solely on $V_C$-maximization performed well in the experiments to be described below.

Back to Independent Component Analysis page.