A hallmark of intelligence is the ability to learn models representing the world at varying levels of abstraction. I will summarize recent work of my group on two approaches to learning multi-scale representations, specifically dynamic abstraction networks and proto-value functions. Dynamic abstraction networks are hierarchical graphical models which combine spatial and temporal abstraction. I will describe approximate inference techniques that exploit the context-specific independences in the hierarchical structure of the model to accelerate learning and reasoning. Proto-value functions are task-independent building blocks of reward-based value functions. Mathematically, they are Fourier eigenfunctions of the Laplace-Beltrami operator on a state space manifold. I will describe a multi-level representation of proto-value functions using diffusion wavelets. Diffusion wavelets construct a multi-level model of stochastic processes on a manifold (graph) using layered spectral analysis of reversible random walks. I show how diffusion wavelets can be used to design a fast policy evaluation algorithm.