Grainger College of Engineering, All Events

View Full Calendar

Amin Sadeghi "WhiteGrad: A fast neural network optimizer"

Event Type
Ceremony/Service
Sponsor
Illinois Computer Science
Virtual
wifi event
Date
Feb 9, 2023   2:00 pm  
Speaker
Dr. Amin Sadeghi, Staff Scientist, Qatar Computer Research Institute
Contact
Candice Steidinger
E-Mail
steidin2@illinois.edu
Views
93
Originating Calendar
Computer Science Speakers Calendar

We look forward to seeing you online on February 9 for the Vision Seminar Series. 

Abstract:

I will present a new neural network optimizer, WhiteGrad, that is faster than SGD and Adam. Unlike Adam and conventional neural network optimizers, WhiteGrad does not operate element-wise. Instead, it uses a form of whitening that involves matrix multiplication. As a result of whitening, learning occurs significantly faster.

Before this work, whitening was rarely used in neural network optimizers, because it is slow. We propose a decomposition technique that makes whitening fast. In short, given a fully connected layer, according to the chain rule, the gradients with respect to the layer weights, are calculated as the product of input activations X and the gradients with respect to the layer outputs G. We propose to whiten X and G before they get multiplied.

In my talk, I will present the intuition and algorithm behind this optimizer. I will also show some promising results on learning of convolutional networks and vision transformers. I will finally discuss a range of new ideas and possibilities that would interest graduate students.

 

Bio: 

Amin Sadeghi completed his PhD under David Forsyth in 2015. He studied efficient object detectors. Then he served as an assistant professor of Computer Science at University of Tehran from 2016 to 2019. Since 2020 he is a staff scientist at Qatar Computer Research Institute studying machine learning and computer vision. He has a track record of multiple successful startups in AI. He has also received research awards including the best student paper award at CVPR.

link for robots only