Outlier Impact: Detection by Consequences

We introduce a novel outlier detection method that utilizes a concrete measure of the statistical impact of an outlier rather than a vague notion of “unusualness”. This allows practitioners to only flag points that materially affect the inferences made from the data while also ensuring these points are anomalous. This is of particular interest in experimentation, where outliers affect the validity of the causal inferences. In particular, we consider a potential outlier’s impact on the mean and the False Positive Rate (FPR) of a hypothesis test.