Dataset Information

Calibration of cognitive tests to address the reliability paradox for decision-conflict tasks.

ABSTRACT: Standard, well-established cognitive tasks that produce reliable effects in group comparisons also lead to unreliable measurement when assessing individual differences. This reliability paradox has been demonstrated in decision-conflict tasks such as the Simon, Flanker, and Stroop tasks, which measure various aspects of cognitive control. We aim to address this paradox by implementing carefully calibrated versions of the standard tests with an additional manipulation to encourage processing of conflicting information, as well as combinations of standard tasks. Over five experiments, we show that a Flanker task and a combined Simon and Stroop task with the additional manipulation produced reliable estimates of individual differences in under 100 trials per task, which improves on the reliability seen in benchmark Flanker, Simon, and Stroop data. We make these tasks freely available and discuss both theoretical and applied implications regarding how the cognitive testing of individual differences is carried out.

SUBMITTER: Kucina T

PROVIDER: S-EPMC10115879 | biostudies-literature | 2023 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Calibration of cognitive tests to address the reliability paradox for decision-conflict tasks.

Kucina Talira T Wells Lindsay L Lewis Ian I de Salas Kristy K Kohl Amelia A Palmer Matthew A MA Sauer James D JD Matzke Dora D Aidman Eugene E Heathcote Andrew A

Nature communications 20230419 1

Standard, well-established cognitive tasks that produce reliable effects in group comparisons also lead to unreliable measurement when assessing individual differences. This reliability paradox has been demonstrated in decision-conflict tasks such as the Simon, Flanker, and Stroop tasks, which measure various aspects of cognitive control. We aim to address this paradox by implementing carefully calibrated versions of the standard tests with an additional manipulation to encourage processing of c ...[more]

PMID: 37076456

Similar Datasets

Project description:UK Biobank is a health resource with data from over 500,000 adults. The cognitive assessment in UK Biobank is brief and bespoke, and is administered without supervision on a touchscreen computer. Psychometric information on the UK Biobank cognitive tests are limited. Despite the non-standard nature of these tests and the limited psychometric information, the UK Biobank cognitive data have been used in numerous scientific publications. The present study examined the validity and short-term test-retest reliability of the UK Biobank cognitive tests. A sample of 160 participants (mean age = 62.59, SD = 10.24) was recruited who completed the UK Biobank cognitive assessment and a range of well-validated cognitive tests ('reference tests'). Fifty-two participants returned 4 weeks later to repeat the UK Biobank tests. Correlations were calculated between UK Biobank tests and reference tests. Two measures of general cognitive ability were created by entering scores on the UK Biobank cognitive tests, and scores on the reference tests, respectively, into separate principal component analyses and saving scores on the first principal component. Four-week test-retest correlations were calculated for UK Biobank tests. UK Biobank cognitive tests showed a range of correlations with their respective reference tests, i.e. those tests that are thought to assess the same underlying cognitive ability (mean Pearson r = 0.53, range = 0.22 to 0.83, p≤.005). The measure of general cognitive ability based on the UK Biobank cognitive tests correlated at r = 0.83 (p < .001) with a measure of general cognitive ability created using the reference tests. Four-week test-retest reliability of the UK Biobank tests were moderate-to-high (mean Pearson r = 0.55, range = 0.40 to 0.89, p≤.003). Despite the brief, non-standard nature of the UK Biobank cognitive tests, some tests showed substantial concurrent validity and test-retest reliability. These psychometric results provide currently-lacking information on the validity of the UK Biobank cognitive tests.

Project description:Neural and behavioral mechanisms during approach-avoidance conflict decision-making are relevant across various psychiatric disorders, particularly anxiety disorders. Studies using approach-avoidance conflict paradigms in healthy adults have identified preliminary neural mechanisms, but findings must be replicated and demonstrated as reliable before further application. This study sought to replicate previous findings and examine test-retest reliability of behavioral (approach behavior, reaction time) and neural (regions of interest [ROIs]) responses during an approach-avoidance conflict task conducted during functional magnetic resonance imaging (fMRI). Thirty healthy adults completed an approach-avoidance conflict task during fMRI on two occasions (mean interval: 17 days; range: 11-32). Effects of task condition during three task phases (decision-making, affective outcome and monetary reward) and intraclass correlation coefficients (ICCs) were calculated across time points. Results replicated that approach behavior was modulated by conflict during decision-making. ROI activations were replicated such that dorsal anterior cingulate cortex (dACC) was modulated by conflict during decision-making, and dACC, striatum, and anterior insula were modulated by valence during affective outcomes (p's <.0083). Approach behavior during conflict demonstrated excellent reliability (ICCs ≥.77). Activation of dACC during conflict decision-making and anterior insula during negative outcomes demonstrated fair reliability (ICCs = .51 and .54), and dACC and striatum activation demonstrated good reliability during negative outcomes (ICCs = .63 and .69). Two additional ROIs (amygdala, left dorsolateral prefrontal cortex) showed good reliability during negative outcomes (ICCs ≥.60). These results characterize several specific behavioral and neuroimaging responses that are replicable and sufficiently reliable during approach-avoidance conflict decision-making to support future utility.

Dataset Information

Calibration of cognitive tests to address the reliability paradox for decision-conflict tasks.

Publications

Calibration of cognitive tests to address the reliability paradox for decision-conflict tasks.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets