Dataset Information

On the number of trials needed to distinguish similar alternatives.

ABSTRACT: A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smaller improvement in quality from the items produced by the current system A. By formalizing this observation, we develop closed-form analytical expressions for the sample efficiency of a number of widely used families of slate-based comparison tests. In empirical trials, these theoretical sample complexity results are shown to be predictive of real-world testing efficiency outcomes. These findings offer opportunities for both more cost-effective testing and a better analytical understanding of the problem.

SUBMITTER: Chierichetti F

PROVIDER: S-EPMC9351503 | biostudies-literature | 2022 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

On the number of trials needed to distinguish similar alternatives.

Chierichetti Flavio F Kumar Ravi R Tomkins Andrew A

Proceedings of the National Academy of Sciences of the United States of America 20220728 31

A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smaller improvement in quality from the items produced by the current system A. By formalizing this observation, we develop closed-form analytical expressions for the sample efficiency of a number of widely ...[more]

PMID: 35901210

Similar Datasets

Project description:ObjectiveThe purpose of this study was to develop a quantitative framework to estimate the likelihood of multifocal epilepsy based on the number of unifocal seizures observed in the epilepsy monitoring unit (EMU).MethodsPatient records from the EMU at Massachusetts General Hospital (MGH) from 2012 to 2014 were assessed for the presence of multifocal seizures as well the presence of multifocal interictal discharges and multifocal structural imaging abnormalities during the course of the EMU admission. Risk factors for multifocal seizures were assessed using sensitivity and specificity analysis. A Kaplan-Meier survival analysis was used to estimate the risk of multifocal epilepsy for a given number of consecutive seizures. To overcome the limits of the Kaplan-Meier analysis, a parametric survival function was fit to the EMU subjects with multifocal seizures and this was used to develop a Bayesian model to estimate the risk of multifocal seizures during an EMU admission.ResultsMultifocal interictal discharges were a significant predictor of multifocal seizures within an EMU admission with a p < 0.01, albeit with only modest sensitivity 0.74 and specificity 0.69. Multifocal potentially epileptogenic lesions on MRI were not a significant predictor p = 0.44. Kaplan-Meier analysis was limited by wide confidence intervals secondary to significant patient dropout and concern for informative censoring. The Bayesian framework provided estimates for the number of unifocal seizures needed to predict absence of multifocal seizures. To achieve 90% confidence for the absence of multifocal seizure, three seizures are needed when the pretest probability for multifocal epilepsy is 20%, seven seizures for a pretest probability of 50%, and nine seizures for a pretest probability of 80%.SignificanceThese results provide a framework to assist clinicians in determining the utility of trying to capture a specific number of seizures in EMU evaluations of candidates for epilepsy surgery.

Project description:ObjectiveSchizophrenia clinical trials commonly measure observed changes in Positive and Negative Syndrome Scale (PANSS) total score. However, it is more intuitive to think of response vs nonresponse, a binary outcome. Assessing binary outcomes enables calculation of number needed to treat (NNT) for therapeutic outcomes, number needed to harm (NNH) for adverse outcomes, and likelihood to be helped or harmed (LHH) to demonstrate benefit/risk tradeoffs. Here, NNT, NNH, and LHH were used to evaluate the clinical usefulness of aripiprazole lauroxil in patients with an acute schizophrenia exacerbation.MethodsCategorical efficacy and tolerability data were taken from the pivotal Phase 3 trial evaluating aripiprazole lauroxil for treatment of an acute exacerbation of schizophrenia. NNT and NNH values, with 95% CIs, were calculated in this post hoc analysis.ResultsUsing the intent-to-treat population for the pooled doses of aripiprazole lauroxil (441 mg [n=196] and 882 mg [n=204] q4w), responder rates (≥30% improvement from baseline PANSS total score) were 35.3% for aripiprazole lauroxil arms vs 18.4% for placebo (n=196), yielding a NNT of 6 (95% CI: 5-11). Discontinuation rates due to adverse events (AEs) were higher among patients randomized to placebo than to either aripiprazole lauroxil dose. Akathisia was the only AE with an incidence ≥5% in each aripiprazole lauroxil group and at least twice that of placebo (11.6%, 11.5%, and 4.3% of the patients receiving aripiprazole lauroxil 441 mg, 882 mg, and placebo, respectively), producing a NNH of 14 (95% CI: 9-33) for pooled aripiprazole lauroxil doses vs placebo. Calculating LHH for therapeutic response vs akathisia, aripiprazole lauroxil was 2.3 times more likely to result in a therapeutic response than an incident of akathisia.ConclusionUsing metrics of NNT, NNH, and LHH, aripiprazole lauroxil was an efficacious and well-tolerated intervention in a pivotal study in patients with an acute schizophrenia exacerbation.

Dataset Information

On the number of trials needed to distinguish similar alternatives.

Publications

On the number of trials needed to distinguish similar alternatives.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets