A simple algorithm that discovers efficient perceptual codes.
Brendan Frey Peter Dayan Geoff Hinton
In M Jenkin & L Harris, editors, Computation and Psychophysical
Mechanisms of Visual Coding, 296-315. Cambridge:
CUP.
Abstract
We describe the ``wake-sleep'' algorithm that allows a multilayer,
unsupervised, neural network to build a hierarchy of representations
of sensory input. The network has bottom-up ``recognition''
connections that are used to convert sensory input into underlying
representations. Unlike most artificial neural networks, it also has
top-down ``generative'' connections that can be used to reconstruct
the sensory input from the representations. In the ``wake'' phase of
the learning algorithm, the network is driven by the bottom-up
recognition connections and the top-down generative connections are
trained to be better at reconstructing the sensory input from the
representation chosen by the recognition process. In the ``sleep''
phase, the network is driven top-down by the generative connections to
produce a fantasized representation and a fantasized sensory input.
The recognition connections are then trained to be better at
recovering the fantasized representation from the fantasized sensory
input. In both phases, the synaptic learning rule is simple and
local. The combined effect of the two phases is to create
representations of the sensory input that are efficient in the
following sense: On average, it takes more bits to describe each
sensory input vector directly than to first describe the
representation of the sensory input chosen by the recognition process
and then describe the difference between the sensory input and its
reconstruction from the chosen representation.
compressed postscript pdf