Recognizing Hand-written Digits Using Hierarchical
products of Experts
Guy Mayraz & Geoffrey Hinton
Gatsby Computational Neuroscience Unit
University College London
17 Queen Square, London WC1N 3AR, UK
The product of experts learning procedure  can discover a set of
stochastic binary features that constitute a non-linear generative model of hand-written
images of digits. The quality of generative models learned in this way can be
assessed by learning a separate model for each class of digit and then comparing the
unnormalized probabilities of test images under the 10 different class-specific models.
To improve discriminative performance, each of the 10 digit models can be given
more layers of feature detectors. The layers are trained sequentially and each layer
learns a generative model of the patterns of feature activities in the preceding layer.
After training, each layer of feature detectors produce a separate, unnormalized
log probability score. With three layers of feature detectors in each of the 10
digit models, a test image produces 30 scores which can be used as inputs to a supervised,
logistic classification network that is trained on separate data. On the MNIST
database, our system is comparable with current state-of-the-art discriminative methods,
demonstrating that the product of experts learning procedure can produce effective
generative models of high-dimensional data.
Download [pdf] [ps.gz]