Beal, M.J., Ghahramani, Z. (2002)The Variational Bayesian EM Algorithm for Incomplete Data: with Application to Scoring Graphical Model Structures
To appear in Bayesian Statistics 7, Oxford University Press, 2003.
preprint: [pdf] [ps.gz]
We present an efficient procedure for estimating the marginal likelihood of probabilistic models with latent variables or incomplete data. This method constructs and optimises a lower bound on the marginal likelihood using variational calculus, resulting in an iterative algorithm which generalises the EM algorithm by maintaining posterior distributions over both latent variables {\it and parameters}. We define the family of conjugate-exponential models---which includes finite mixtures of exponential family models, factor analysis, hidden Markov models, linear state-space models, and other models of interest---for which this bound on the marginal likelihood can be computed very simply through a modification of the standard EM algorithm. In particular, we focus on applying these bounds to the problem of scoring discrete directed graphical model structures (Bayesian networks). Extensive simulations comparing the variational bounds to the usual approach based on the Bayesian Information Criterion (BIC) and to a sampling-based gold standard method known as Annealed Importance Sampling (AIS) show that variational bounds substantially outperform BIC in finding the correct model structure at relatively little computational cost, while approaching the performance of the much more costly AIS procedure. Using AIS allows us to provide the first serious case study of the tightness of variational bounds. We also analyse the perfomance of AIS through a variety of criteria, and outline directions in which this work can be extended.
Beal, M.J., Attias, A. and Jojic, N. (2002)Audio-Video Sensor Fusion with Probabilistic Graphical Models
To be presented at the European Conference on Computer Vision (ECCV), May 2002.
We present a new approach to modeling and processing multimedia data. This approach is based on graphical models that combine audio and video variables. We demonstrate it by developing a new algorithm for tracking a moving object in a cluttered, noisy scene using two microphones and a camera. Our model uses unobserved variables to describe the data in terms of the process that generates them. It is therefore able to capture and exploit the statistical structure of the audio and video data separately, as well as their mutual dependencies. Model parameters are learned from data via an EM algorithm, and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location from data. We demonstrate successful performance on multimedia clips captured in real world scenarios using off-the-shelf equipment.
Beal, M.J., Jojic, N. and Attias, A. (2002)A Self-Calibrating Algorithm for Speaker Tracking Based on Audio-Visual Statistical Models
Accepted for lecture presentation at the International Conference on Acoustics Speech and Signal Processing (ICASSP), May 2002.
[pdf] [ps.gz]
We present a self-calibrating algorithm for audio-visual tracking using two microphones and a camera. The algorithm uses a parametrized statistical model which combines simple models of video and audio. Using unobserved variables, the model describes the process that generates the observed data. Hence, it is able to capture and exploit the statistical structure of the audio and video data, as well as their mutual dependencies. The model parameters are estimated by the EM algorithm; object templates are learned and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location using the model. Successful performance is demonstrated on real multimedia clips.
Beal, M.J., Ghahramani, Z. and Rasmussen, C.E. (2002)The Infinite Hidden Markov Model
In Advances in Neural Information Processing Systems 14, eds. T. Dietterich, S. Becker, Z. Ghahramani, MIT Press, 2002.
[pdf] [ps.gz]
We show that it is possible to extend hidden Markov models to have a countably infinite number of hidden states. By using the theory of Dirichlet processes we can implicitly integrate out the infinitely many transition parameters, leaving only three hyperparameters which can be learned from data. These three hyperparameters define a hierarchical Dirichlet process capable of capturing a rich set of transition dynamics. The three hyperparameters control the time scale of the dynamics, the sparsity of the underlying state-transition matrix, and the expected number of distinct hidden states in a finite sequence. In this framework it is also natural to allow the alphabet of emitted symbols to be infinite---consider, for example, symbols being possible words appearing in English text.
Ghahramani, Z. and Beal, M.J. (2001)Propagation Algorithms for Variational Bayesian Learning
In Advances in Neural Information Processing Systems 13, eds. T.K. Leen, T. Dietterich, V. Tresp, MIT Press, 2001.
[pdf] [ps.gz] [poster] [software]
Variational approximations are becoming a widespread tool for Bayesian learning of graphical models. We provide some theoretical results for the variational updates in a very general family of {\em conjugate-exponential} graphical models. We show how the belief propagation and the junction tree algorithms can be used in the inference step of variational Bayesian learning. Applying these results to the Bayesian analysis of linear-Gaussian state-space models we obtain a learning procedure that exploits the Kalman smoothing propagation, while integrating over all model parameters. We demonstrate how this can be used to infer the hidden state dimensionality of the state-space model in a variety of synthetic problems and one real high-dimensional data set.
Variational Inference for Bayesian Mixtures of Factor Analysers
In Advances in Neural Information Processing Systems 12:449-455, eds. S. A. Solla, T.K. Leen, K. Müller, MIT Press, 2000.
[pdf] [ps.gz] [poster] [software]
We present an algorithm that infers the model structure of a mixture
of factor analysers using an efficient and deterministic variational
approximation to full Bayesian integration over model parameters. This
procedure can automatically determine the optimal number of components
and the local dimensionality of each component (i.e.\ the number of
factors in each factor analyser). Alternatively it can be used to
infer posterior distributions over number of components and
dimensionalities. Since all parameters are integrated out the method
is not prone to overfitting. Using a stochastic procedure for
adding components it is possible to perform the variational
optimisation incrementally and to avoid local maxima. Results show
that the method works very well in practice and correctly infers the
number and dimensionality of nontrivial synthetic examples.
By importance sampling from the variational approximation we show how
to obtain unbiased estimates of the true evidence, the exact
predictive density, and the KL divergence between the variational
posterior and the true posterior, not only in this model but for
variational approximations in general.
Graphical Models and Variational Methods
Book chapter in Advanced Mean Field methods - Theory and Practice, eds. D. Saad and M. Opper, MIT Press. jacket
Part of this work is that discussed at NIPS*99 workshop of the same title as the book.
[pdf] [ps.gz]
We review the use of variational methods of approximating inference and learning in probabilistic graphical models. In particular, we focus on variational approximations to the integrals required for Bayesian learning. For models in the conjugate-exponential family, a generalisation of the EM algorithm is derived that iterates between optimising hyperparameters of the distribution over parameters, and inferring the hidden variable distributions. These approximations make use of available propagation algorithms for probabilistic graphical models. We give two case studies of how the variational Bayesian approach can be used to learn model structure: inferring the number of clusters and dimensionalities in a mixture of factor analysers, and inferring the dimension of the state space of a linear dynamical system. Finally, importance sampling corrections to the variational approximations are discussed, along with their limitations.