Machine Learning Seminar: Dr. Lisha Chen, "Preference-Guided Model Steering to Explore Trade-offs on the Pareto Front."

Abstract: When a learning system balances competing objectives, such as helpfulness and safety in an LLM, how can it offer trade-offs that reflect users’ preferences? A common approach is to vary the weights assigned to each objective. However, the distribution of objective weights often does not match users’ desired distribution of model performance on the Pareto front (PF).
To address this mismatch, I will first introduce SURF, an analytically tractable method for bi-objective problems. It targets settings where users want a diverse menu of trade-offs, ideally uniform on the PF. Our analysis shows that the mismatch arises because, as the objective weight varies, the solutions trace the PF at a non-uniform speed. The arc-length map records the distance traveled along the front as a function of the weight; by estimating and inverting it, SURF selects weights that yield uniformly distributed points on the PF. SURFACE extends this idea to higher-dimensional PFs, general distributions beyond the uniform one, and general scalarization methods. With more than two objectives, arc length generalizes to the area or volume of regions on the PF, and SURFACE generalizes SURF by estimating how regions in weight space map to regions of different volumes on the PF. Finally, I will discuss applications to multi-objective reinforcement learning, LLM alignment, and multi-task speech LLMs, where adjusting objective weights during training or fine-tuning helps steer a model toward users’ preferred trade-offs among objectives or tasks.

Bio: Lisha Chen is an Assistant Professor of Electrical and Computer Engineering at the University of Rochester. Her research spans machine learning, optimization, and statistical signal and image processing, with a current focus on multi-objective and multi-level optimization and learning. She earned her Ph.D. and M.S. from Rensselaer Polytechnic Institute and her B.S. from Huazhong University of Science and Technology. She has delivered tutorials at ICASSP and AAAI and received the IEEE Signal Processing Society Scholarship and the IBM AIRC Fellowship.