The methods described in this paper are primarily devoted to finding binary or at least quasi-binary codes. Each code symbol participating in a quasi-binary code is either 0 or 1 in response to a given input pattern or emits a constant value in response to every input pattern. Therefore, binary codes are a special case of quasi-binary codes. Most of our quasi-binary codes will be created by starting out from real-valued codes.
Recall that there are three criteria that a binary factorial code must fulfill:
1. The binary criterion: Each code-symbol should be either 1 or 0 in response to a given input pattern.
2. The invertibility criterion: It must be possible to reconstruct the input from the code. In cases where the environment is too complex (or too noisy) to be fully coded into limited internal representations (i.e., in the case of binary codes where there are more than input patterns), we want to relax the invertibility criterion. In that case, we still want the internal representations to convey maximal information about the inputs. The focus of this paper, however, is on situations like the ones studied in (Barlow et. al, 1989): Noise-free environments and sufficient representational capacity in the representational units. In the latter case, reversibility is equivalent to Infomax à la Linsker (1988).
3. The independence criterion: The occurrence of each code symbol
ought to be independent of all other code symbols. If the
binary criterion is fulfilled, then
we may rewrite
the independence criterion by requiring that