Learning invariant feature hierarchies for vision is a major challenge for Deep Learning. I will discuss three different strategies for learning invariant features in an unsupervised, semi-supervised, or purely supervised manner: unsupervised methods based on sparse auto-encoders, unsupervised methods based on sparse autoencoder with feature pooling layers to create invariances, methods based on product networks that separate the "what" from the "where", semi-supervised methods that use similarity cues, and purely supervised methods with architectural constraints to help them learn with few samples. The good news is that a simple unsupervised sparse auto-encoder method followed by supervised refinement can match the performance and "hard-wired" systems based on SIFT and SVM with specialized kernels on object recognition tasks with very few labeled training samples. The bad news is that deep systems with essentially random filters can do almost as well, provided that the architecture contains the "right" set of non-linearities. Moreover, learning-based system do perform considerably better than random ones on larger datasets, and significantly better than hard-wired ones on specialized images datasets, such as MNIST.
BACK
