Title: Near-Optimal Non-Parametric Sequential Tests and Confidence Sequences with Possibly Dependent Observations
Abstract: Sequential tests and their implied confidence sequences, which are valid at arbitrary stopping times, promise flexible statistical inference and on-the-fly decision making. However, strong guarantees are limited to parametric sequential tests, which can suffer high type-I error rates in practice because reality isn't parametric, or to concentration-bound-based sequences, which are overly conservative so we get wide intervals and take too long to detect effects. We consider a classic delayed-start normal-mixture sequential probability ratio test and provide the first asymptotic (in the delay) analysis under general non-parametric data generating processes. We guarantee that type-I-error rates approach a user-specified α-level (primarily by leveraging a martingale strong invariance principle). Moreover, we show that the expected time-to-reject approaches the minimum possible among all α-level tests (primarily by leveraging a new identity inspired by discretizing Itô's lemma). Together, our results establish these (ostensibly parametric) tests as general-purpose, non-parametric, and near-optimal. We illustrate this via numerical experiments and a retrospective re-analysis of A/B tests at Netflix. Time permitting, I will discuss the application to inference using data collected adaptively by a contextual bandit algorithm.
Papers: Will mostly talk about https://arxiv.org/abs/2212.14411
Time permitting will also discuss https://arxiv.org/abs/2106.00418 and https://arxiv.org/abs/2106.01723
Bio: Nathan Kallus is an Associate Professor at the Cornell Tech campus of Cornell University in NYC and a Research Director at Netflix. Nathan's research interests include the statistics of optimization under uncertainty, causal inference especially when combined with machine learning, sequential and dynamic decision making, and algorithmic fairness. He holds a PhD in Operations Research from MIT as well as a BA in Mathematics and a BS in Computer Science from UC Berkeley. Before coming to Cornell, Nathan was a Visiting Scholar at USC's Department of Data Sciences and Operations and a Postdoctoral Associate at MIT's Operations Research and Statistics group.
