Bayesian Decision Theory for Evaluating AI Systems
Abstract: This talk reviews Bayesian decision theory and shows how it can be used to understand AI systems. Decision problems can be found throughout AI systems. A decision problem maps actions of an agent and states of nature to payoffs. Information, aka a signal, about the state can potentially be used to improve decisions. The rational benchmark performance in a decision problem is defined as the expected payoff of the Bayes-optimal action given the signal. Without the signal, the rational baseline is defined as the expected payoff of the optimal action given the prior distribution over states. The value of information is the difference between the benchmark and baseline. Performance of parts of an AI system can be evaluated in terms how much of the value of information is obtained. Moreover, scrutinizing the decision theoretic framework enables sources of losses to be identified. This talk applies this framework to understand reliance in human-AI decision systems.
Biography:
Prof. Hartline’s research introduces design and analysis methodologies from computer science to understand and improve outcomes of economic, legal, and AI systems. Optimal behavior and outcomes in complex environments are complex and, therefore, should not be expected; instead, the theory of approximation can show that simple and natural behaviors are approximately optimal in complex environments. This approach is applied to auction theory and mechanism design in his graduate textbook Mechanism Design and Approximation which is under preparation.
Prof. Hartline received his Ph.D. in 2003 from the University of Washington under the supervision of Anna Karlin. He was a postdoctoral fellow at Carnegie Mellon University under the supervision of Avrim Blum; and subsequently a researcher at Microsoft Research in Silicon Valley. He joined Northwestern University in 2008 where he is a professor of computer science. He was on sabbatical at Harvard University in the Economics Department during the 2014 calendar year and visiting Microsoft Research, New England for the Spring of 2015. He was on sabbatical at Stanford University for the 2023-2024 academic year.
Prof. Hartline is the director of Northwestern’s Online Markets Lab, he was a founding codirector of the Institute for Data, Econometrics, Algorithms, and Learning from 2019-2022, and is a cofounder of virtual conference organizing platform Virtual Chair.