9th May 2005 — Iterative fitting and scaling
Iain covered “Iterative Proportional Fitting” (IPF) and “Generalized Iterative Scaling” (GIS).
Original papers: GIS, “Generalized Iterative Scaling for Log-Linear Models” J. N. Darroch; D. Ratcliff The Annals of Mathematical Statistics, Vol. 43, No. 5. (Oct., 1972), pp. 1470-1480.; IPF, “On a least squares adjustment of a sampled frequency table when the expected marginal totals are known” WE Deming, FF Stephan, Annals of Mathematical Statistics, 1940.
Less painful references: Sam Roweis’s lecture notes, Adam Berger’s notes.
Russ Salakhutdinov has papers with Sam Roweis and Zoubin on improving iterative bound optimization algorithms and comparing to gradient methods.
Tom Minka runs comparisons of many algorithms including GIS variants on logistic regression.
We alluded to Yee Whye Teh and Max Welling’s Unified propagation and scaling. Is this an area where intelligent message passing can beat gradient methods?