CS Compiler Seminar: Matei Budiu, "Nautilus: An Auto-Scheduling Tensor Compiler for Efficient Tiled GPU Kernels."

- Sponsor
- Siebel School of Computing and Data Science
- Speaker
- Matei Budiu
- Contact
- SSCDS Area Support
- agk@illinois.edu
- Views
- 58
- Originating Calendar
- Siebel School Speakers Calendar
Abstract: We present Nautilus, a machine learning compiler that automatically generates efficient GPU kernels for attention from a math-like specification. Nautilus’s key insight is that the optimizations needed to bridge math and kernels can be separated at the tile level: existing tile compilers already optimize well at below the tile level, so Nautilus uses them as backends and schedules only optimizations above the tile level, drastically shrinking the search space. Nautilus’s auto-scheduler automates advanced reduction fusion, a non-local transformation, while keeping programs in the regular form tile compilers expect, and its lowering pipeline introduces a virtual-register tile IR that enables expression rewriting across loop iterations. Composing these transformations, Nautilus derives FlashAttention-3-style kernels from a linear-algebraic description of attention, and discovers different strategies for other workloads, such as a Flash-Decoding-style kernel for long decode contexts, rather than instantiating a fixed template. Across five transformer models on NVIDIA GH200 and RTX 5090 GPUs, Nautilus achieves a geometric-mean speedup of 1.18× on GH200 and 1.20× on RTX 5090 over state-of-the-art compilers and libraries, and is on par with hand-optimized cuDNN kernels.
Speaker Bio.: Matei Budiu is a second year Ph.D. student in the Siebel School of Computing and Data Science at the University of Illinois, Urbana-Champaign (UIUC), advised by Prof. Sasa Misailovic. His research interests lie in the fields of compilers, machine learning, parallel computing, graphics, and edge-device systems. His current work on the auto-scheduler component of Nautilus has just been submitted to MLSys 2027.
In Summer of 2026, Matei was an applied science intern at Amazon Lab126's System Architecture group, where he worked on a high-level simulator for running machine learning models on an NPU. Matei received a Bachelor's degree from Carnegie Mellon University's School of Computer Science, where he focused on computer graphics and systems.