Abstract: Operator fusion has become a key optimization for deep learning, which combines multiple deep learning operators to improve data reuse and reduce global memory transfers. However, existing tensor compilers struggle to fuse complex reduction computations involving loop-carried dependencies, such as attention mechanisms.
The paper introduces Neptune, a tensor compiler for advanced operator fusion for sequences of reduction operators. Neptune presents a new approach for advanced operator fusion, which intentionally breaks some existing dependencies and compensates by constructing algebraic correction expressions that allow the kernel to produce the correct result. On ten attention-based benchmarks, Neptune, starting from simple attention code and a high-level scheduling template, outperforms existing compilers like Triton, TVM, and FlexAttention, including Triton-based implementations of FlashAttention. Across four different GPU architectures from NVIDIA and AMD, Neptune-generated kernels have an average speedup of 1.35x over the next best alternative, demonstrating its effectiveness for deep learning workloads.
Bio: Yifan Zhao is a CS Ph.D. student at the University of Illinois Urbana-Champaign (UIUC), advised by Prof. Sasa Misailovic and Prof. Vikram Adve. Yifan's current research focuses on tensor compilers, which leverages domain-specific optimizations, accuracy-aware optimizations, and autotuning, to make deep learning models and applications more efficient.