Recognizing Hand-written Digits Using Hierarchical products of Experts

Guy Mayraz & Geoffrey Hinton
Gatsby Computational Neuroscience Unit
University College London
17 Queen Square, London WC1N 3AR, UK

Abstract

The product of experts learning procedure [1] can discover a set of stochastic binary features that constitute a non-linear generative model of hand-written images of digits.  The quality of generative models learned in this way can be assessed by learning a separate model for each class of digit and then comparing the unnormalized probabilities of test images under the 10 different class-specific models.   To improve discriminative performance, each of the 10 digit models can be given more layers of feature detectors.  The layers are trained sequentially and each layer learns a generative model of the patterns of feature activities in the preceding layer.   After training, each layer of feature detectors produce a separate, unnormalized log probability score.  With three layers of feature detectors in each of the 10 digit models, a test image produces 30 scores which can be used as inputs to a supervised, logistic classification network that is trained on separate data.  On the MNIST database, our system is comparable with current state-of-the-art discriminative methods, demonstrating that the product of experts learning procedure can produce effective generative models of high-dimensional data.

---

Download  [pdf] [ps.gz]