Safe Reinforcement Learning in Healthcare Decision Making
Yonatan Mintz
Assistant Professor, Department of Industrial and Systems Engineering
University of Wisconsin- Madison
Abstract: One of the key challenges in sequential decision making is optimizing systems safely in the case of partial information. While much of the existing work has focused on addressing this challenge in the case of either partially known states or partially known system dynamics, it is further exacerbated in cases where both states and dynamics are partially known. For instance the setting of computing heparin doses for patients fits this paradigm since the concentration of heparin in the patient cannot be measured directly and the rates at which patients metabolize it vary greatly between individuals. While many approaches proposed to resolve the challenge in this setting are model free, they require complex models that are not transparent to decision makers, and are difficult to analyze and guarantee safety. However, if some of the structure of the dynamics is known, a model based approach can be leveraged to provide safe policies with practical empirical performance and theoretical worst case guarantees. In this talk we propose a model based framework to address the challenge of partially observed states and dynamics in the context of designing personalized doses of heparin. We use a predictive model based on pharmacokinetics (the study of how the body effects substances through absorption, distribution, and metabolism) parameterized individually by patient, and infer the current concentration of heparin and predict future therapeutic effects taking into account different patients' characteristics. We formulate the patient parameter estimation problem in to a mixed integer linear program and show that our estimates are statistically consistent. We leverage this model by developing an adaptive dosing algorithm that outputs asymptotically optimal dose sequences based on a scenario generation approach, this approach is also capable of ensuring that the required heparin doses are maintained within a safe level. We validate our models with numerical experiments by first comparing the predictive capabilities of our model against existing machine learning techniques and demonstrating how our dosing algorithm can keep patients' related medical tests within a therapeutic range in a simulated ICU environment. Our results show that our methods are capable of maintaining patients in therapeutic range for 87.7% of the treatment time as opposed to existing weight based protocols that can only do so for 55.6% of the treatment time.
Biography: Yonatan Mintz is an assistant professor in the Industrial and Systems Engineering department at the University of Wisconsin, Madison. His research focuses on the application of machine learning and automated decision making to human sensitive contexts. One application of his research has been on using patient level data, to create precision interventions . Yonatan is also interested in the sociotechnical implications of machine learning algorithms and has done work on fairness, accountability, and transparency in automated decision making. In terms of methodology his research explores topics in machine learning theory, stochastic control, reinforcement learning, and nonconvex optimization. Yonatan's work has been recognized as a finalist in the INFORMS Health Applications Society Pierskalla Paper competition, a best poster award from the NeurIPS joint workshop on AI for Social Good, and he has been actively invited to publicly speak about his work in both print and televised media including PBS. His research has been funded by multiple awards from the National Institutes of Health (NIH) and American Family Insurance. Prior to joining UW--Madison, Yonatan was a postdoctoral research fellow at the department of Industrial and Systems Engineering at the Georgia Institute of Technology. Yonatan received his B.S. in Industrial and Systems Engineering with a concentration in Operations Research from Georgia Tech in 2012, and his Ph.D. in Industrial Engineering and Operations Research from the University of California, Berkeley in 2018.