Probabilistic and Unsupervised Learning
Approximate Inference and Learning in Probabilistic Models
(2017)

Dates 2 October - 14 December 2017
Lectures Mondays and Thursdays 11:00-13:00 (note any exceptions below)
Tutorials Wednesdays 9:00-11:00
Lecturer Maneesh Sahani
Coordinator (CS) Dmitry Adamsky. Please direct all correspondence regarding coursework, extensions and administrative issues to Dmitry.
TAs (CS) Harshil Shah, Raza Habib, Benoit Gaujac; (Gatsby/SWC) Jorge Menendez Canelas, Kirsty McNaught, Lea Duncker, Michael Arbel, Wenkai Xu
Location Ground Floor Seminar Room, 25 Howland Street, London W1T 4JG
About the course

This course (offered as two successive modules to MSc students) provides an in-depth introduction to statistical modelling, unsupervised, and some supervised learning techniques. It presents probabilistic approaches to modelling and their relation to coding theory and Bayesian statistics. A variety of latent variable models will be covered including mixture models (used for clustering), dimensionality reduction methods, time series models such as hidden Markov models which are used in speech recognition and bioinformatics, Gaussian process models, independent components analysis, hierarchical models, and nonlinear models. The course will present the foundations of probabilistic graphical models (e.g. Bayesian networks and Markov networks) as an overarching framework for unsupervised modelling. We will cover Markov chain Monte Carlo sampling methods and variational approximations for inference. Time permitting, students will also learn about other topics in probabilistic (or Bayesian) machine learning.

The complete course forms a component of the Gatsby PhD programme, and is mandatory for Gatsby students. The two component modules are also available to students on the UCL MSc in Machine Learning and UCL MSc in Computational Statistics and Machine Learning. The first part (GI18: Probabilistic and Unsupervised Learning) may be used to fill a core requirement in each Masters programme. The second part (GI16: Approximate Inference and Learning in Probabilistic Models) is an optional module, but is only available to students who have completed GI18.

Mailing list

If you plan to attend the lectures, please sign up to join the email list here. (If you used the previous version -- we do have your email, no need to sign up again!)

General attendance

UCL Students, postdocs and faculty from outside the unit and these programmes are welcome to attend the lectures as long as space is available. If you are not a student on either MSc programme above, please do not register for the course without consulting the instructor.

Prerequisites

A good background in statistics, calculus, linear algebra, and computer science. You should thoroughly review the maths in the following cribsheet [pdf] [ps] before the start of the course. This quiz may help you check where you stand. You must know Matlab, Octave or Python/NumPy.

Text

There is no required textbook. However, the following are excellent sources for many of the topics covered here.

David J.C. MacKay (2003) Information Theory, Inference, and Learning Algorithms, Cambridge University Press.
Christopher M. Bishop (2006) Pattern Recognition and Machine Learning, Springer-Verlag.
David Barber (2012) Bayesian Reasoning and Machine Learning, Cambridge University Press.
Kevin Patrick Murphy (2012) Machine Learning: a Probabilistic Perspective, MIT Press.
Daphne Koller and Nir Friedman (2009) Probabilistic Graphical Models, MIT Press. (This contains a more extensive treatment of graphical models, good for reference)

Some of our work will depend numerical computation. A very useful reference in this area is:

Gene H. Golub and Charles F. Van Loan (2013) Matrix Computations, 4th ed, Johns Hopkins Univ. Press
Reading material

Specific recommendations for additional reading can be found here. New material may be assembled as course progresses.

Lecture schedule and slides (provisional) Lecture slides and assignments will be posted here as the course progresses. Note that the content and sequence of lectures may vary slightly from those of previous years. Some lectures (boldface dates) will be moved to the Wednesday morning tutorial slot.

 

GI18
2/10: Introduction to Probabilistic Learning / screen version 5/10: Intro contd.
9/10: Latent Variables / screen version 11/10: contd.
16/10: Expectation Maximisation / screen version 19/10: Latent chain models / screen version
23/10: Latent chain models contd. 26/10: Graphical models
30/10: contd 2/11: Model selection and GPs
break
GI16
13/11: Intractable models 16/11: contd.
20/11: Variational approximations 23/11: Expectation Propagation
27/11: Belief Propagation 30/11: Convexity-based Methods
4/12: Sampling 7/12: contd.
11/12: TBD 14/12: TBD
Assignments
50% of the total mark (for each module) is based on coursework assignments (the other 50% being based on the final written examination).

Assignments (coursework) are due at the beginning of lecture on the date specified (except Assignement 6, which is due at 5pm). They may be handed in by email or on paper to the course coordinator (for CS/ML MSc students) or to a Gatsby TA (for Gatsby and SWC students). Late assignments (including those handed in after the beginning of lecture) will be penalised.

Late Assignment Policy: we will be using the official UCL-wide late policy.

Provided you submit credible attempts to all the required questions on time, you may submit late answers to the BONUS questions subject to a penalty that is the lesser of:

  • 10 marks up to 2 days late
  • 50% of the maximum mark on the late work up to 5 days late
  • 100% of the marks earned (this is to protect you from losing marks by attempting the bonus questions late!)
Assignment 0 - maths quiz to be discussed in tutorial 4 Oct 2017
Assignment 1        [binarydigits.txt,  bindigit.m,  bindigit.py] due 16 Oct 2017
Assignment 2        [geyser.txt ssm_spins.txt ssm_spins_test.txt ssm_kalman.m]--> due 30 Oct 2017
Assignment 3 due 20 Nov 2017
Assignments 4 and 5 due 11 Dec 2017
Assignment 6 due 8 Jan 2017
To attend Please see the note above