Researchers Cut Uncertainty in Messy Real-World Experiments
A new statistical method lets researchers get tighter, more reliable confidence intervals when running experiments with interference and complex designs—conditions that plague A/B tests and field studies in practice. The approach guarantees conservative estimates while potentially cutting safety margins in half, saving companies money on larger-than-necessary sample sizes.
Originaltitel: Optimized Variance Estimation under Interference and Complex Experimental Designs
Unbiased and consistent variance estimators generally do not exist for design-based treatment effect estimators because experimenters never observe more than one potential outcome for any unit. The problem is exacerbated by interference and complex experimental designs. Experimenters must accept conservative variance estimators in these settings, but they can strive to minimize the conservativeness. In this article, we show that the task of constructing a minimally conservative variance estimator can be interpreted as an optimization problem that aims to find the lowest estimable upper bound of the true variance given the experimenter’s risk preferences and knowledge of the potential outcomes. We characterize the set of admissible bounds in the class of quadratic forms, and we demonstrate that the optimization problem is a convex program for many natural objectives. The resulting variance estimators are guaranteed to be conservative regardless of whether the background knowledge used to construct the bound is correct, but the estimators are less conservative if the provided information is reasonably accurate. Numerical results show that the resulting variance estimators can be considerably less conservative than existing estimators, allowing experimenters to draw more informative inferences about treatment effects. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.