Factor Analysis using Delta-Rule Wake-Sleep
Learning.
Radford Neal Peter Dayan
Neural Computation, 9, 1781-1803.
Abstract
We describe a linear network that models correlations between
real-valued visible variables using one or more real-valued hidden
variables - a factor analysis model. This model can be seen
as a linear version of the ``Helmholtz machine'', and its parameters
can be learned using the ``wake-sleep'' method, in which learning of
the primary ``generative'' model is assisted by a ``recognition''
model, whose role is to fill in the values of hidden variables based
on the values of visible variables. The generative and recognition
models are jointly learned in ``wake'' and ``sleep'' phases, using just
the delta rule. This learning procedure is comparable in simplicity to
Hebbian learning, which produces a somewhat different
representation of correlations in terms of principal components. We
argue that the simplicity of wake-sleep learning makes factor analysis
a plausible alternative to Hebbian learning as a model of
activity-dependent cortical plasticity.
compressed postscript pdf