|
DECISION AND CONTROL LECTURE THE GRAINGER COLLEGE OF ENGINEERING |
| TITLE | The emergence of clusters in self-attention dynamics |
Sponsor | Decision and Control |
Date | Thursday, October 5, 2023 |
Time | 4:00 PM |
LOCATION | Coordinated science lab, rm b02 |
Speaker: Postdoctoral Associate, Borjan Geshkovski from MIT |
ABSTRACT |
With remarkable empirical success, Transformers enable large language models to compute very powerful representations using the self-attention mechanism. We model this mechanism as an interacting particle systems to brings and demonstrate the formation of clusters as the number of layers goes to infinity. Based on joint work with Cyril Letrouit (CNRS), Yury Polyanskiy (MIT) and Philippe Rigollet (MIT). |
BIO |
Borjan Geshkovski is currently a postdoc at MIT Math, where he works with Philippe Rigollet. He got his PhD from the Autonomous University in Madrid under the supervision of Enrique Zuazua. His research interests are centred around control, learning and PDE. |
