Can expected error costs justify testing a hypothesis at multiple alpha levels rather than searching for an elusive optimal alpha?

ABSTRACT: Simultaneous testing of one hypothesis at multiple alpha levels can be performed within a conventional Neyman-Pearson framework. This is achieved by treating the hypothesis as a family of hypotheses, each member of which explicitly concerns test level as well as effect size. Such testing encourages researchers to think about error rates and strength of evidence in both the statistical design and reporting stages of a study. Here, we show that these multi-alpha level tests can deliver acceptable expected total error costs. We first present formulas for expected error costs from single alpha and multiple alpha level tests, given prior probabilities of effect sizes that have either dichotomous or continuous distributions. Error costs are tied to decisions, with different decisions assumed for

SUBMITTER: Aisbett J 

PROVIDER: S-EPMC11424007 | biostudies-literature | 2024

REPOSITORIES: biostudies-literature