Seminarios Académicos

Abstract: Online platforms rely on experiments (A/B tests) to aid decision-making. However, prior work has shown that in marketplace experiments, interactions between users can create interference effects that lead to biased estimates of the treatment effect. We develop mathematical models to capture these interference effects and study the biases that arise. We show that the magnitude of the treatment effect bias depends on the level of supply and demand imbalance in the platform. Building on these insights, we propose a novel class of experimental designs using “two-sided randomization” (TSR) that reduces bias across wide ranges of market imbalance. We then consider the effect of interference on the resulting platform decisions. We show that a second type of bias arises that also impacts decisions. Specifically, interference also leads to biased estimates of the standard error, which can result in confidence intervals that are too wide or too narrow, causing the platform to be under or over-confident in their decisions. We show that there are interaction effects between the standard error and treatment effect biases that impact the quality of decisions. Finally, we outline work to assess the impacts on decisions made in the Airbnb marketplace.