Belief networks, or directed probabilistic graphical models, are a powerful representation for modeling complex probability distributions. Learning the structure of a belief network, particularly one with hidden units, has been difficult, however. The Indian buffet process (IBP) is a distribution on sparse binary matrices which has previously been used as a nonparametric Bayesian prior on the directed structure of a belief network with a single infinitely wide hidden layer. I will review the IBP and present very recent work introducing the cascading Indian buffet process (CIBP), which provides a nonparametric Bayesian prior on the structure of a layered, directed belief network that is both infinitely deep and infinitely wide. The CIBP results in networks in which a provably finite yet unbounded set of hidden units contribute to the distribution over visible units. We use the CIBP as the prior on structure of a continuous sigmoidal belief network so that each unit can vary its behavior between discrete and continuous states. This flexible model thus allows inference of the number of hidden units, the directed edge structure between the units, the depth of the network, and the appropriate representation for each unit. We provide Markov chain Monte Carlo algorithms for inference and perform exploratory analysis of the sparse structure and unit modalities arising from a popular data set.
Joint work with Ryan P. Adams (Cambridge) and Hanna M. Wallach (U Mass Amherst).
BACK
