The technique known in the machine learning community as "backpropagation" is a special case of a method known as "reverse-mode automatic differentiation", or "reverse AD". We will explore forward and reverse AD using a novel formulation based on differential geometry. In this formulation, they can be naturally generalized to apply to a much broader range of computer programs, including programs that contain iterate-to-fixedpoint loops, and programs that invoke or embody higher-order functions, programs that invoke optimizers, or even programs which themselves invoke AD operators. Techniques like fast exact Hessian-vector multiplication, Pineda/Almeida fixedpoint backpropagation, and a wide variety of other techniques can be defined and implemented as one-liners using these generalized AD operators. These method allow very complicated systems, like bi-level optimization architectures, to be built and optimized using gradient methods. The method has been formalized using the tools of modern Programming Language Theory, and a research prototype implementation has been constructed which exhibits startling good numeric performance.
(Joint work with Jeffrey Mark Siskind)
BACK
