Better-than-chance classification for signal detection.

ABSTRACT: The estimated accuracy of a classifier is a random quantity with variability. A common practice in supervised machine learning, is thus to test if the estimated accuracy is significantly better than chance level. This method of signal detection is particularly popular in neuroimaging and genetics. We provide evidence that using a classifier's accuracy as a test statistic can be an underpowered strategy for finding differences between populations, compared to a bona fide statistical test. It is also computationally more demanding than a statistical test. Via simulation, we compare test statistics that are based on classification accuracy, to others based on multivariate test statistics. We find that the probability of detecting differences between two distributions is lower for accuracy-bas

SUBMITTER: Rosenblatt JD 

PROVIDER: S-EPMC8036001 | biostudies-literature | 2021 Apr

REPOSITORIES: biostudies-literature