EXAMPLE 2: Text Compression

The example from the previous section was based on artificial data from a stochastic automaton. Can neural predictors offer something for redundancy reduction in natural language? How do they compare to standard data compression algorithms?

The method [28] reviewed in this section is an instance of a strategy known as ``predictive coding'' or ``model-based coding''. A neural predictor network $P$ is trained to approximate the conditional probability distribution of possible characters, given the previous characters. $P$'s outputs are fed into the Arithmetic Coding algorithm (e.g. [41]) that generates short codes for characters with low information content (characters with high predicted probability) and long codes for characters conveying a lot of information (highly unpredictable characters) [28].