Feature to prototype transition in neural networks


March 31, 2017 - 10:30am - 11:30am
Northwest Building Room 425
About the Speaker
Dmitry Krotov
Speaker Title: 
Post-doctoral Fellow
Speaker Affiliation: 
Institute for Advanced Study in Princeton

I will describe unusual models of associative memory with higher order (higher than quadratic) interactions, and their relationship to neural networks used in deep learning. Associative memory is conventionally described by feedback neural networks with dynamical convergence to stable points. Deep learning typically uses feed-forward neural nets without dynamics. However, a simple duality relates these two different views when applied to problems of pattern classification. From the perspective of associative memory such models deserve attention because they make it possible to store a much larger number of memories, compared to the quadratic case. In the dual description, these models correspond to feed-forward neural networks with one hidden layer and unusual activation functions transmitting the activities of the visible neurons to the hidden layer. These activation functions are rectified polynomials of a higher degree rather than the rectified linear functions used
in deep learning. The network learns representations of the data in terms of features for rectified linear functions, but as the power in the activation function is increased there is a gradual shift to a prototype-based representation, the two extreme regimes of pattern recognition known in cognitive psychology. Theoretical arguments support the existence of this transition, which will be illustrated by using the MNIST data set of handwritten digits.