What Is the Test-Retest Reliability of Common Task-Functional MRI Measures? New Empirical Evidence and a Meta-Analysis.
Ontology highlight
ABSTRACT:
SUBMITTER: Elliott ML
PROVIDER: S-EPMC7370246 | biostudies-literature | 2020 Jul
REPOSITORIES: biostudies-literature
Publications
What Is the Test-Retest Reliability of Common Task-Functional MRI Measures? New Empirical Evidence and a Meta-Analysis.
Elliott Maxwell L ML Knodt Annchen R AR Ireland David D Morris Meriwether L ML Poulton Richie R Ramrakha Sandhya S Sison Maria L ML Moffitt Terrie E TE Caspi Avshalom A Hariri Ahmad R AR
Psychological science 20200603 7
Identifying brain biomarkers of disease risk is a growing priority in neuroscience. The ability to identify meaningful biomarkers is limited by measurement reliability; unreliable measures are unsuitable for predicting clinical outcomes. Measuring brain activity using task functional MRI (fMRI) is a major focus of biomarker development; however, the reliability of task fMRI has not been systematically evaluated. We present converging evidence demonstrating poor reliability of task-fMRI measures. ...[more]
Similar Datasets
Project description:IntroductionFunctional MRI (fMRI) is commonly used to investigate the neural mechanisms underlying psychological processes and behavioral responses. However, to draw well-founded conclusions from fMRI studies, more research on the reliability of fMRI is needed.MethodsWe invited a sample of 41 female students to participate in two identical fMRI sessions, separated by 5 weeks on average. To investigate the potential effect of left-handedness on the stability of neural activity, we oversampled left-handed participants (N = 20). Inside the scanner, we presented photographs of familiar and unfamiliar children's faces preceded by neutral and threatening primes to the participants. We calculated intraclass correlations (ICCs) to investigate the test-retest reliability of peak activity in areas that showed significant activity during the first session (primary visual cortex, fusiform face area, inferior frontal gyrus, and superior temporal gyrus). In addition, we examined how many trials were needed to reliably measure the effects.ResultsAcross all participants, only fusiform face area activity in response to faces showed good test-retest reliability (ICC = 0.71). All other test-retest reliabilities were low (0.01 ≤ ICC ≤ 0.35). Reliabilities varied only slightly with increasing numbers of trials, with no consistent increase in ICCs. Test-retest reliabilities for left-handed participants (0.28 ≤ ICC ≤0.66) were generally somewhat higher than for right-handed participants (-0.13 ≤ ICC ≤0.75), but not statistically significant.ConclusionOur study shows good test-retest reliability for fusiform facer area activity in response to faces, but low test-retest reliability for other contrasts and areas.
| S-EPMC6422824 | biostudies-literature
Project description:BackgroundFunctional MRI (fMRI) tasks are increasingly being used to advance knowledge of the etiology and maintenance of obesity and eating disorders. Thus, understanding the test-retest reliability of BOLD signal contrasts from these tasks is important.ObjectivesTo evaluate test-retest reliability of responses in reward-related brain regions to food receipt paradigms (palatable tastes, anticipated palatable tastes), food picture paradigms (high-calorie food pictures), a monetary reward paradigm (winning money and anticipating winning money), and a thin female model picture paradigm (thin female model pictures).MethodWe conducted secondary univariate contrast-based analyses in data drawn from 4 repeated-measures fMRI studies. Participants (Study 1: N = 60, mean [M] age = 15.2 ± 1.1 y; Study 2: N = 109, M age = 15.1 ± 0.9 y; Study 3: N = 39, M age = 21.2 ± 3.7 y; Study 4: N = 62, M age = 29.7 ± 6.2 y) completed the same tasks over 3-wk to 3-y test-retest intervals. Studies 3 and 4 included participants with eating disorders and obesity, respectively.ResultsTest-retest reliability of the food receipt and food picture paradigms was poor, with average ICC values ranging from 0.07 to 0.20. The monetary reward paradigm and the thin female model picture paradigm also showed poor test-retest reliability: average ICC values 0.21 and 0.12, respectively. Although several regions demonstrated moderate to good test-retest reliability, these results did not replicate across studies using similar paradigms. In Studies 3 and 4, but not Study 1, test-retest reliability in visual processing regions was moderate to good when contrasting single conditions with a low-level baseline.ConclusionsResults underscore the importance of examining the temporal reliability of fMRI tasks and call for the development and use of well-validated standardized fMRI tasks in eating- and obesity-related studies that can provide more reliable measures of neural activation. The trials were registered at clinicaltrials.gov as NCT02084836, NCT01949636, NCT03261050, and NCT03375853.
| S-EPMC8326039 | biostudies-literature
Project description:Recognizing others' emotional expressions is vital for socioemotional development; impairments in this ability occur in several psychiatric disorders. Further study is needed to map the development of this ability and to evaluate its components as potential transdiagnostic endophenotypes. Before doing so, however, research is required to substantiate the test-retest reliability of scores of the face emotion identification tasks linked to developmental psychopathology. The current study estimated test-retest reliability of scores of one such task, the facial expression labeling task (FELT) among a sample of twin children (N = 157; ages 9-14). Participants completed the FELT at two visits two to five weeks apart. Participants discerned the emotion presented of faces depicting six emotions (i.e., happiness, anger, sadness, fear, surprise, and disgust) morphed with a neutral face to provide 10 levels of increasing emotional expressivity. The present study found strong test-retest reliability (Pearson r) of the FELT scores across all emotions. Results suggested that data from this task may be effectively analyzed using a latent growth curve model to estimate overall ability (i.e., intercept; r's = 0.76-0.85) and improvement as emotions become clearer (i.e., linear slope; r's = 0.69-0.83). Evidence of high test-retest reliability of this task's scores informs future developmental research and the potential identification of transdiagnostic endophenotypes for child psychopathology. (PsycINFO Database Record
| S-EPMC5568997 | biostudies-literature
Project description:Vergence eye movements are the inward and outward rotation of the eyes responsible for binocular coordination. While studies have mapped and investigated the neural substrates of vergence, it is not well understood whether vergence eye movements evoke the blood oxygen level-dependent signal reliably in separate experimental visits. The test-retest reliability of stimulus-induced vergence eye movement tasks during a functional magnetic resonance imaging (fMRI) experiment is important for future randomized clinical trials (RCTs). In this study, we established region of interest (ROI) masks for the vergence neural circuit. Twenty-seven binocularly normal young adults participated in two functional imaging sessions measured on different days on the same 3T Siemens scanner. The fMRI experiments used a block design of sustained visual fixation and rest blocks interleaved between task blocks that stimulated eight or four vergence eye movements. The test-retest reliability of task-activation was assessed using the intraclass correlation coefficient (ICC), and that of spatial extent was assessed using the Dice coefficient. Functional activation during the vergence eye movement task of eight movements compared to rest was repeatable within the primary visual cortex (ICC = 0.8), parietal eye fields (ICC = 0.6), supplementary eye field (ICC = 0.5), frontal eye fields (ICC = 0.5), and oculomotor vermis (ICC = 0.6). The results demonstrate significant test-retest reliability in the ROIs of the vergence neural substrates for functional activation magnitude and spatial extent using the stimulus protocol of a task block stimulating eight vergence eye movements compared to sustained fixation. These ROIs can be used in future longitudinal RCTs to study patient populations with vergence dysfunctions.
| S-EPMC7186292 | biostudies-literature
Project description:ObjectiveTo examine the feasibility and test-retest reliability of encoding-task functional magnetic resonance imaging (fMRI) in mild Alzheimer disease (AD).DesignRandomized, double-blind, placebo-controlled study.SettingMemory clinical trials unit.ParticipantsWe studied 12 patients with mild AD (mean [SEM] Mini-Mental State Examination score, 24.0 [0.7]; mean Clinical Dementia Rating score, 1.0) who had been taking donepezil hydrochloride for more than 6 months from the placebo arm of a larger 24-week study (n = 24, 4 scans on weeks 0, 6, 12, and 24, respectively).InterventionsPlacebo and 3 face-name, paired-associate encoding, block-design blood oxygenation level-dependent fMRI scans in 12 weeks.Main outcome measuresWe performed whole-brain t maps (P < .001, 5 contiguous voxels) and hippocampal regions-of-interest analyses of extent (percentage of active voxels) and magnitude (percentage of signal change) for novel-greater-than-repeated face-name contrasts. We also calculated intraclass correlation coefficients and power estimates for hippocampal regions of interest.ResultsTask tolerability and data yield were high (95 of 96 scans yielded favorable-quality data). Whole-brain maps were stable. Right and left hippocampal regions-of-interest intraclass correlation coefficients were 0.59 to 0.87 and 0.67 to 0.74, respectively. To detect 25.0% to 50.0% changes in week-0 to week-12 hippocampal activity using left-right extent or right magnitude with 80.0% power (2-sided α = .05) requires 14 to 51 patients. Using left magnitude requires 125 patients because of relatively small signal to variance ratios.ConclusionsEncoding-task fMRI was successfully implemented in a single-site, 24-week, AD randomized controlled trial. Week 0 to 12 whole-brain t maps were stable, and test-retest reliability of hippocampal fMRI measures ranged from moderate to substantial. Right hippocampal magnitude may be the most promising of these candidate measures in a leveraged context. These initial estimates of test-retest reliability and power justify evaluation of encoding-task fMRI as a potential biomarker for signal of effect in exploratory and proof-of-concept trials in mild AD. Validation of these results with larger sample sizes and assessment in multisite studies is warranted.
| S-EPMC3291175 | biostudies-literature
Project description:Research in perception and attention has typically sought to evaluate cognitive mechanisms according to the average response to a manipulation. Recently, there has been a shift toward appreciating the value of individual differences and the insight gained by exploring the impacts of between-participant variation on human cognition. However, a recent study suggests that many robust, well-established cognitive control tasks suffer from surprisingly low levels of test-retest reliability (Hedge, Powell, & Sumner, 2018b). We tested a large sample of undergraduate students (n = 160) in two sessions (separated by 1-3 weeks) on four commonly used tasks in vision science. We implemented measures that spanned a range of perceptual and attentional processes, including motion coherence (MoCo), useful field of view (UFOV), multiple-object tracking (MOT), and visual working memory (VWM). Intraclass correlations ranged from good to poor, suggesting that some task measures are more suitable for assessing individual differences than others. VWM capacity (intraclass correlation coefficient [ICC] = 0.77), MoCo threshold (ICC = 0.60), UFOV middle accuracy (ICC = 0.60), and UFOV outer accuracy (ICC = 0.74) showed good-to-excellent reliability. Other measures, namely the maximum number of items tracked in MOT (ICC = 0.41) and UFOV number accuracy (ICC = 0.48), showed moderate reliability; the MOT threshold (ICC = 0.36) and UFOV inner accuracy (ICC = 0.30) showed poor reliability. In this paper, we present these results alongside a summary of reliabilities estimated previously for other vision science tasks. We then offer useful recommendations for evaluating test-retest reliability when considering a task for use in evaluating individual differences.
| S-EPMC9344221 | biostudies-literature
Project description:While structural network analysis consolidated the hypothesis of cerebral small vessel disease (SVD) being a disconnection syndrome, little is known about functional changes on the level of brain networks. In patients with genetically defined SVD (CADASIL, n = 41) and sporadic SVD (n = 46), we independently tested the hypothesis that functional networks change with SVD burden and mediate the effect of disease burden on cognitive performance, in particular slowing of processing speed. We further determined test-retest reliability of functional network measures in sporadic SVD patients participating in a high-frequency (monthly) serial imaging study (RUN DMC-InTENse, median: 8 MRIs per participant). Functional networks for the whole brain and major subsystems (i.e., default mode network, DMN; fronto-parietal task control network, FPCN; visual network, VN; hand somatosensory-motor network, HSMN) were constructed based on resting-state multi-band functional MRI. In CADASIL, global efficiency (a graph metric capturing network integration) of the DMN was lower in patients with high disease burden (standardized beta = -.44; p [corrected] = .035) and mediated the negative effect of disease burden on processing speed (indirect path: std. beta = -.20, p = .047; direct path: std. beta = -.19, p = .25; total effect: std. beta = -.39, p = .02). The corresponding analyses in sporadic SVD showed no effect. Intraclass correlations in the high-frequency serial MRI dataset of the sporadic SVD patients revealed poor test-retest reliability and analysis of individual variability suggested an influence of age, but not disease burden, on global efficiency. In conclusion, our results suggest that changes in functional connectivity networks mediate the effect of SVD-related brain damage on cognitive deficits. However, limited reliability of functional network measures, possibly due to age-related comorbidities, impedes the analysis in elderly SVD patients.
| S-EPMC7294060 | biostudies-literature
Project description:Despite their wide-spread use, only limited information is available on the comparative test-retest reliability of task-based functional and resting state magnetic resonance imaging measures of blood oxygen level dependence (tb-fMRI and rs-fMRI) and cerebral blood flow (CBF) using arterial spin labeling. This information is critical to designing properly powered longitudinal studies. Here we comprehensively quantified and compared the test-retest reliability and reproducibility performance of 8 commonly applied fMRI tasks, 6 rs-fMRI metrics and CBF in 30 healthy volunteers. We find large variability in test-retest reliability performance across the different tb-fMRI paradigms and rs-fMRI metrics, ranging from poor to excellent. A larger extent of activation in tb-fMRI is linked to higher between-subject reliability of the respective task suggesting that differences in the amount of activation may be used as a first reliability estimate of novel tb-fMRI paradigms. For rs-fMRI, a good reliability of local activity estimates is paralleled by poor performance of global connectivity metrics. Evaluated CBF measures provide in general a good to excellent test-reliability matching or surpassing the best performing tb-fMRI and rs-fMRI metrics. This comprehensive effort allows for direct comparisons of test-retest reliability between the evaluated MRI domains and measures to aid the design of future tb-fMRI, rs-fMRI and CBF studies.
| S-EPMC6224062 | biostudies-literature
Project description:BackgroundThe combination of resting-state functional MRI (R-fMRI) technique and graph theoretical approaches has emerged as a promising tool for characterizing the topological organization of brain networks, that is, functional connectomics. In particular, the construction and analysis of high-resolution brain connectomics at a voxel scale are important because they do not require prior regional parcellations and provide finer spatial information about brain connectivity. However, the test-retest reliability of voxel-based functional connectomics remains largely unclear.AimsThis study tended to investigate both short-term (∼20 min apart) and long-term (6 weeks apart) test-retest (TRT) reliability of graph metrics of voxel-based brain networks.MethodsBased on graph theoretical approaches, we analyzed R-fMRI data from 53 young healthy adults who completed two scanning sessions (session 1 included two scans 20 min apart; session 2 included one scan that was performed after an interval of ∼6 weeks).ResultsThe high-resolution networks exhibited prominent small-world and modular properties and included functional hubs mainly located at the default-mode, salience, and executive control systems. Further analysis revealed that test-retest reliabilities of network metrics were sensitive to the scanning orders and intervals, with fair to excellent long-term reliability between Scan 1 and Scan 3 and lower reliability involving Scan 2. In the long-term case (Scan 1 and Scan 3), most network metrics were generally test-retest reliable, with the highest reliability in global metrics in the clustering coefficient and in the nodal metrics in nodal degree and efficiency.ConclusionWe showed high test-retest reliability for graph properties in the high-resolution functional connectomics, which provides important guidance for choosing reliable network metrics and analysis strategies in future studies.
| S-EPMC6493187 | biostudies-literature
Project description:Best practices are currently being developed for the acquisition and processing of resting-state magnetic resonance imaging data used to estimate brain functional organization-or "functional connectivity." Standards have been proposed based on test-retest reliability, but open questions remain. These include how amount of data per subject influences whole-brain reliability, the influence of increasing runs versus sessions, the spatial distribution of reliability, the reliability of multivariate methods, and, crucially, how reliability maps onto prediction of behavior. We collected a dataset of 12 extensively sampled individuals (144 min data each across 2 identically configured scanners) to assess test-retest reliability of whole-brain connectivity within the generalizability theory framework. We used Human Connectome Project data to replicate these analyses and relate reliability to behavioral prediction. Overall, the historical 5-min scan produced poor reliability averaged across connections. Increasing the number of sessions was more beneficial than increasing runs. Reliability was lowest for subcortical connections and highest for within-network cortical connections. Multivariate reliability was greater than univariate. Finally, reliability could not be used to improve prediction; these findings are among the first to underscore this distinction for functional connectivity. A comprehensive understanding of test-retest reliability, including its limitations, supports the development of best practices in the field.
| S-EPMC6248395 | biostudies-literature