Approaches to analyzing binary data for large-scale A/B testing

ABSTRACT: An industry-academic collaboration was established to evaluate the choice of statistical test and study design for A/B testing in larger-scale industry experiments. Specifically, the standard approach at the industry partner was to apply a t-test for all outcomes, both continuous and binary, and to apply naïve interim monitoring strategies that had not evaluated the potential implications on operating characteristics such as power and type I error rates. Although many papers have summarized the robustness of the t-test, its performance for the A/B testing context of large-scale proportion data, with or without interim analyses, is needed. Investigating the effect of interim analyses on the robustness of the t-test is important, because interim analyses rely on a fraction of the total sampl

SUBMITTER: Zhou W 

PROVIDER: S-EPMC9982610 | biostudies-literature | 2023 Feb

REPOSITORIES: biostudies-literature