The Bayesian analysis of neural networks is difficult because a simple prior over weights implies a complex prior over functions. We investigate the use of a Gaussian process prior over functions, which permits the predictive Bayesian analysis for fixed values of hyperparameters to be carried out exactly using matrix operations. Two methods, using optimization and averaging (via Hybrid Monte Carlo) over hyperparameters have been tested on a number of challenging problems and have produced excellent results.
Advances in Neural Information Processing Systems 8, eds. D. S. Touretzky, M. C. Mozer, M. E. Hasselmo, MIT Press, 1996, pp. 598-604
Available as ps.