Regret in Treatment Choice when Welfare varies with an Uncertain Event: The Prediction-Threshold Problem

We study the maximum regret (MR) of binary treatment choice in a population with observed covariates x, when welfare varies with an uncertain binary event. We study decision making with plug-in probabilistic predictions of the event and pre-specified decision thresholds, which we term the prediction-threshold problem. In this setting, the optimal treatment for persons with covariate value x is B if the conditional probability P(y = 1|x) of a binary outcome y exceeds a particular x-specific threshold and is A otherwise. This structure is common in medical decision making and also arises in non-medical contexts such as criminal justice. Plug-in prediction uses data to estimate P(y|x) and acts as if the estimate is accurate. However, plug-in prediction is often performed with misspecified prediction models and conventional x-invariant thresholds. We aim to shed new light on MR when treatment choice is performed this way. We use a combination of algebraic and computational analysis of limit and sample MR, demonstrating how MR depends on the prediction model, the state space, and the thresholds used to choose treatments.