We look forward to seeing you online on February 9 for the Vision Seminar Series.
I will present a new neural network optimizer, WhiteGrad, that is faster than SGD and Adam. Unlike Adam and conventional neural network optimizers, WhiteGrad does not operate element-wise. Instead, it uses a form of whitening that involves matrix multiplication. As a result of whitening, learning occurs significantly faster.
Before this work, whitening was rarely used in neural network optimizers, because it is slow. We propose a decomposition technique that makes whitening fast. In short, given a fully connected layer, according to the chain rule, the gradients with respect to the layer weights, are calculated as the product of input activations X and the gradients with respect to the layer outputs G. We propose to whiten X and G before they get multiplied.
In my talk, I will present the intuition and algorithm behind this optimizer. I will also show some promising results on learning of convolutional networks and vision transformers. I will finally discuss a range of new ideas and possibilities that would interest graduate students.
Amin Sadeghi completed his PhD under David Forsyth in 2015. He studied efficient object detectors. Then he served as an assistant professor of Computer Science at University of Tehran from 2016 to 2019. Since 2020 he is a staff scientist at Qatar Computer Research Institute studying machine learning and computer vision. He has a track record of multiple successful startups in AI. He has also received research awards including the best student paper award at CVPR.