DNN-based encoding models for the visual cortex fail to generalize out of the training data distribution.
Spandan Madan1, Will Xiao1, Mingran Cao2, Hanspeter Pfister1, Margaret Livingstone1, Gabriel Kreiman1;
1Harvard University, 2The Francis Crick Institute
We characterize the generalization capabilities of DNN-based encoding models when predicting neuronal responses from the primate visual ventral stream. Using a large-scale dataset of neuronal responses from the macaque IT cortex to over 11,000 images, we investigate the impact of distribution shifts on neural activity by dividing the images into multiple training and Out-Of-Distribution (OOD) test sets. This includes different types of OOD domain shifts in the form of image contrast, hue, intensity, and semantic object categories. Overall, we find models performed much worse at predicting neuronal responses for out-of-distribution images, dipping to as low as 20% of the performance on in-distribution test images. Furthermore, we found that this generalization performance under OOD shifts can be well accounted by an image similarity metric—the cosine distance between image representations extracted from a pre-trained object recognition model is a strong predictor of neural predictivity under different distribution shifts (R^2=-0.76).