I gave a talk in Cognitive & Neural Computation Lab at Yale!

Link to Cognitive & Neural Computation Lab (Director Ilker Yildirim)

Title:

The Dynamic Alignment of Ventral Stream Neuronal Tuning with Generative Object Manifold

Abstract:
Traditional studies of visual neural coding typically involve designing stimuli, recording neuronal responses, and iteratively refining the stimuli until a simple yet effective one elicits a strong response. Recently, however, neuroscientists (e.g., Ponce, Xiao, et al. 2019) have automated this process using generative models (such as GANs) and optimizers in a closed-loop procedure called Evolution. This method synthesizes and optimizes images based on neuronal responses until stimuli that maximize activation are found.

Despite its success across various brain regions from V1 to PFC (Wang, Ponce, 2022; Rose, Ponce, 2024), a key question remains: Why does neuronal tuning effectively guide the sampling of latent codes in generative models, and is this true for all such models? We hypothesize that there is a form of neuron-generator alignment—that is, if the complexity of the composition of neuronal tuning and generative mapping is too high, the neuron-guided optimization will fail.

To test this hypothesis, we applied neuron-guided Evolution in two generative models: DeePSim, which parametrizes general textural patterns, and BigGAN, which parametrizes object-centric images, while recording neurons along the ventral stream (Wang, Ponce, 2025). We found that, higher up the visual hierarchy, neurons became increasingly easy to guide evolution in BigGAN space, while increasingly hard for DeePSim space, though both performed similarly in pIT. Moreover, response dynamics revealed that optimization in BigGAN recruited later neuronal responses in IT, suggesting that representations aligned with the object manifold emerge later. Despite these global differences, neurons directed the synthesis of similar local features in both generative spaces. In summary, our results suggest that via dynamics, IT neurons act as effective “inverse functions” for the object generative model, thereby enhancing their steerability on the BigGAN image manifold.