Targeted test evaluation: a framework for designing diagnostic accuracy studies with clear study hypotheses.
Ontology highlight
ABSTRACT:
SUBMITTER: Korevaar DA
PROVIDER: S-EPMC6921417 | biostudies-literature | 2019
REPOSITORIES: biostudies-literature
Publications
Targeted test evaluation: a framework for designing diagnostic accuracy studies with clear study hypotheses.
Korevaar Daniël A DA Gopalakrishna Gowri G Cohen Jérémie F JF Bossuyt Patrick M PM
Diagnostic and prognostic research 20191219
Most randomized controlled trials evaluating medical interventions have a pre-specified hypothesis, which is statistically tested against the null hypothesis of no effect. In diagnostic accuracy studies, study hypotheses are rarely pre-defined and sample size calculations are usually not performed, which may jeopardize scientific rigor and can lead to over-interpretation or "spin" of study findings. In this paper, we propose a strategy for defining meaningful hypotheses in diagnostic accuracy st ...[more]
Similar Datasets
Project description:The number of diagnostic test accuracy (DTA) studies concerning biomarkers have gradually increased during the past years. However, study designs remain imperfect, and the statistical methods used are not meaningful in some published studies. Here, we introduce recommendations for designing DTA studies, including consecutive enrollment of participants with uniform inclusion and exclusion criteria, blinded testing and interpretation, prespecified thresholds, and the use of one reference standard for all subjects. In addition, we also describe more relevant statistical methods in DTA studies, including decision curve analysis (DCA), nomograms, diagnostic model and scale, net reclassification index (NRI), and the integrated discriminatory index (IDI). This review may help clinicians to better design DTA studies that investigating biomarkers.
| S-EPMC6989996 | biostudies-literature
Project description:Comparative accuracy studies evaluate the relative performance of two or more diagnostic tests. As any other form of research, such studies should be reported in an informative manner, to allow replication and to be useful for decision-making. In this study we aimed to assess whether and how components of test comparisons were reported in comparative accuracy studies. We evaluated 100 comparative accuracy studies, published in 2015, 2016 or 2017, randomly sampled from 238 comparative accuracy systematic reviews. We extracted information on 20 reporting items, pertaining to the identification of the test comparison, its validity, and the actual results of the comparison. About a third of the studies (n = 36) did not report the comparison as a study objective or hypothesis. Although most studies (n = 86) reported how participants had been allocated to index tests, we could often not evaluate whether test interpreters had been blinded to the results of other index tests (n = 40; among 59 applicable studies), nor could we identify the sequence of index tests (n = 52; among 90 applicable studies) or the methods for comparing measures of accuracy (n = 59). Two-by-four table data (revealing the agreement between index tests) were only reported by 9 of 90 paired comparative studies. More than half of the studies (n = 64) did not provide measures of statistical imprecision for comparative accuracy. Our findings suggest that components of test comparisons are frequently missing or incompletely described in comparative accuracy studies included in systematic reviews. Explicit guidance for reporting comparative accuracy studies may facilitate the production of full and informative study reports.
| S-EPMC8246725 | biostudies-literature
Project description:ObjectiveMachine learning (ML) diagnostic tools have significant potential to improve health care. However, methodological pitfalls may affect diagnostic test accuracy studies used to appraise such tools. We aimed to evaluate the prevalence and reporting of design characteristics within the literature. Further, we sought to empirically assess whether design features may be associated with different estimates of diagnostic accuracy.Materials and methodsWe systematically retrieved 2 × 2 tables (n = 281) describing the performance of ML diagnostic tools, derived from 114 publications in 38 meta-analyses, from PubMed. Data extracted included test performance, sample sizes, and design features. A mixed-effects metaregression was run to quantify the association between design features and diagnostic accuracy.ResultsParticipant ethnicity and blinding in test interpretation was unreported in 90% and 60% of studies, respectively. Reporting was occasionally lacking for rudimentary characteristics such as study design (28% unreported). Internal validation without appropriate safeguards was used in 44% of studies. Several design features were associated with larger estimates of accuracy, including having unreported (relative diagnostic odds ratio [RDOR], 2.11; 95% confidence interval [CI], 1.43-3.1) or case-control study designs (RDOR, 1.27; 95% CI, 0.97-1.66), and recruiting participants for the index test (RDOR, 1.67; 95% CI, 1.08-2.59).DiscussionSignificant underreporting of experimental details was present. Study design features may affect estimates of diagnostic performance in the ML diagnostic test accuracy literature.ConclusionsThe present study identifies pitfalls that threaten the validity, generalizability, and clinical value of ML diagnostic tools and provides recommendations for improvement.
| S-EPMC7647361 | biostudies-literature
Project description:In 1996, shortly after the founding of The Cochrane Collaboration, leading figures in test evaluation research established a Methods Group to focus on the relatively new and rapidly evolving methods for the systematic review of studies of diagnostic tests. Seven years later, the Collaboration decided it was time to develop a publication format and methodology for Diagnostic Test Accuracy (DTA) reviews, as well as the software needed to implement these reviews in The Cochrane Library. A meeting hosted by the German Cochrane Centre in 2004 brought together key methodologists in the area, many of whom became closely involved in the subsequent development of the methodological framework for DTA reviews. DTA reviews first appeared in The Cochrane Library in 2008 and are now an integral part of the work of the Collaboration.
| S-EPMC3851548 | biostudies-literature
Project description:BackgroundSeveral types of statistical methods are currently available for the meta-analysis of studies on diagnostic test accuracy. One of these methods is the Bivariate Model which involves a simultaneous analysis of the sensitivity and specificity from a set of studies. In this paper, we review the characteristics of the Bivariate Model and demonstrate how it can be extended with a discrete latent variable. The resulting clustering of studies yields additional insight into the accuracy of the test of interest.MethodsA Latent Class Bivariate Model is proposed. This model captures the between-study variability in sensitivity and specificity by assuming that studies belong to one of a small number of latent classes. This yields both an easier to interpret and a more precise description of the heterogeneity between studies. Latent classes may not only differ with respect to the average sensitivity and specificity, but also with respect to the correlation between sensitivity and specificity.ResultsThe Latent Class Bivariate Model identifies clusters of studies with their own estimates of sensitivity and specificity. Our simulation study demonstrated excellent parameter recovery and good performance of the model selection statistics typically used in latent class analysis. Application in a real data example on coronary artery disease showed that the inclusion of latent classes yields interesting additional information.ConclusionsOur proposed new meta-analysis method can lead to a better fit of the data set of interest, less biased estimates and more reliable confidence intervals for sensitivities and specificities. But even more important, it may serve as an exploratory tool for subsequent sub-group meta-analyses.
| S-EPMC4105799 | biostudies-literature
Project description:ObjectiveTo provide guidance on standards for reporting studies of diagnostic test accuracy for dementia disorders.MethodsAn international consensus process on reporting standards in dementia and cognitive impairment (STARDdem) was established, focusing on studies presenting data from which sensitivity and specificity were reported or could be derived. A working group led the initiative through 4 rounds of consensus work, using a modified Delphi process and culminating in a face-to-face consensus meeting in October 2012. The aim of this process was to agree on how best to supplement the generic standards of the STARD statement to enhance their utility and encourage their use in dementia research.ResultsMore than 200 comments were received during the wider consultation rounds. The areas at most risk of inadequate reporting were identified and a set of dementia-specific recommendations to supplement the STARD guidance were developed, including better reporting of patient selection, the reference standard used, avoidance of circularity, and reporting of test-retest reliability.ConclusionSTARDdem is an implementation of the STARD statement in which the original checklist is elaborated and supplemented with guidance pertinent to studies of cognitive disorders. Its adoption is expected to increase transparency, enable more effective evaluation of diagnostic tests in Alzheimer disease and dementia, contribute to greater adherence to methodologic standards, and advance the development of Alzheimer biomarkers.
| S-EPMC4115600 | biostudies-literature
Project description:Limiting the search date is a common approach utilised in therapeutic/interventional rapid reviews. Yet the accuracy of pooled estimates is unknown when applied to rapid reviews of diagnostic test accuracy studies. Data from all systematic reviews of diagnostic test accuracy studies published in the Cochrane Database of Systematic Reviews, until February 2022 were collected. Meta-analyses with at least five studies were included to emulate rapid reviews by limiting the search to the recent 1, 2, 3, 4, 5, 10, 15, 20, 25, 30, 35 and 40 years. The magnitude of the pooled area under the curve (AUC), sensitivity and specificity for the full meta-analysis and the rapid reviews were compared. A total of 846 diagnostic meta-analyses were included. When the search date was limited to the recent 10 and 15 years, more than 75% and 80% of meta-analyses presented less than 5% difference between the pooled AUC, sensitivity and specificity of the full meta-analysis and the rapid review. There was little gain in the precision of the pooled estimates when the emulated rapid reviews included more than 15 years in the search. Rapid reviews restricted by search date are a valid and reliable approach for diagnostic test accuracy studies. Robust evidence can be achieved by restricting the search date to the recent 10-15 years. Future studies need to examine the reduction in workload and time to finish the rapid reviews under different search date limits.
| S-EPMC9922791 | biostudies-literature
Project description:ObjectivesThe objective of this study is to introduce methods to use all of the information without omission when individual studies provide multiple effect sizes according to multiple cut-off values (thresholds) during diagnostic test accuracy (DTA) for data integration. For diagnostic test meta-analysis, a general performance method for synthesizing data according to one cut value in one study and a performance method for synthesizing data according to two or more cut values in one study were compared and analyzed.MethodsAs sample data for meta-analysis of DTA studies, 13 DTA studies on prostate cancer (34 effect sizes including total cut-offs) were collected. The summary statistics were calculated and the summary line was analyzed using the "meta", "mada", and "diagmeta" packagesof the R software.ResultsThe summary statistics of the random effect model univariate analysis of the "meta" package with a single cut-off corresponding to the highest Youden index in a single study and those of the bivariate analysis of the "mada" package were highly similar. However, in the bivariate analysis of the "diagmeta" package including all cut-off values, the sensitivity decreased and the specificity increased as the amount of data increased.ConclusionsConsidering the heterogeneity of the summary receiver op erating characteristic curve and the use of all given cut-offs, the use of the bivariate analysis model of the "diagmeta" package is recommended. This study focused on practical methods of DTA rather than theoretical concepts for use by researchers whose fields of study are non-statistics related. By performing this study, we hope that many researchers will use R software to determine the DTA more easily, and that there will be greater interest in related research.
| S-EPMC10106543 | biostudies-literature
Project description:ImportanceSystematic reviews of medical imaging diagnostic test accuracy (DTA) studies are affected by between-study heterogeneity due to a range of factors. Failure to appropriately assess the extent and causes of heterogeneity compromises the interpretability of systematic review findings.ObjectiveTo assess how heterogeneity has been examined in medical imaging DTA studies.Evidence reviewThe PubMed database was searched for systematic reviews of medical imaging DTA studies that performed a meta-analysis. The search was limited to the 40 journals with highest impact factor in the radiology, nuclear medicine, and medical imaging category in the InCites Journal Citation Reports of 2021 to reach a sample size of 200 to 300 included studies. Descriptive analysis was performed to characterize the imaging modality, target condition, type of meta-analysis model used, strategies for evaluating heterogeneity, and sources of heterogeneity identified. Multivariable logistic regression was performed to assess whether any factors were associated with at least 1 source of heterogeneity being identified in the included meta-analyses. Methodological quality evaluation was not performed. Data analysis occurred from October to December 2022.FindingsA total of 242 meta-analyses involving a median (range) of 987 (119-441 510) patients across a diverse range of disease categories and imaging modalities were included. The extent of heterogeneity was adequately described (ie, whether it was absent, low, moderate, or high) in 220 studies (91%) and was most commonly assessed using the I2 statistic (185 studies [76%]) and forest plots (181 studies [75%]). Heterogeneity was rated as moderate to high in 191 studies (79%). Of all included meta-analyses, 122 (50%) performed subgroup analysis and 87 (36%) performed meta-regression. Of the 242 studies assessed, 189 (78%) included 10 or more primary studies. Of these 189 studies, 60 (32%) did not perform meta-regression or subgroup analysis. Reasons for being unable to investigate sources of heterogeneity included inadequate reporting of primary study characteristics and a low number of included primary studies. Use of meta-regression was associated with identification of at least 1 source of variability (odds ratio, 1.90; 95% CI, 1.11-3.23; P = .02).Conclusions and relevanceIn this systematic review of assessment of heterogeneity in medical imaging DTA meta-analyses, most meta-analyses were impacted by a moderate to high level of heterogeneity, presenting interpretive challenges. These findings suggest that, despite the development and availability of more rigorous statistical models, heterogeneity appeared to be incomplete, inconsistently evaluated, or methodologically questionable in many cases, which lessened the interpretability of the analyses performed; comprehensive heterogeneity assessment should be addressed at the author level by improving personal familiarity with appropriate statistical methodology for assessing heterogeneity and involving biostatisticians and epidemiologists in study design, as well as at the editorial level, by mandating adherence to methodologic standards in primary DTA studies and DTA meta-analyses.
| S-EPMC10905313 | biostudies-literature
Project description:BackgroundTo investigate the accuracy of screening tests for gestational diabetes mellitus (GDM) in Southeast Asian pregnant women.MethodsWe searched PubMed (MEDLINE), Web of Science, Cochrane Library, ClinicalTrials.gov, Google Scholar, and Google for relevant articles published in English up to November 2018 using search terms related to GDM, screening tests for GDM and diagnostic performance. The studies were independently screened and selected by both authors. The methodological quality of the included studies was independently assessed by quality assessment of diagnostic accuracy studies 2. A hierarchical summary receiver operating characteristic (HSROC) model was created to estimate the HSROC curve. The summary sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and diagnostic odds ratio were calculated in a meta-analysis using bivariate random-effects model.ResultsA total of 19 studies were included in which the 100 g oral glucose tolerance test (OGTT) and 75 g OGTT were the two common reference standards for diagnosis of GDM. Most points of diagnostic performance in the HSROC 50 g GCT curve compared with the 100 g OGTT reference standard were clustered in the upper left-hand quadrant. The pooled sensitivity and specificity of the 50 g GCT were 79% (95% confidence interval [CI] 64%-89%) and 74% (95% CI 59%-85%), respectively. For the 75 g OGTT reference standard, the non-fasting 2-hour plasma glucose showed quite similar sensitivity the 50 g GCT compared with the 100 g OGTT reference standard. The pooled sensitivities and specificities of the fasting plasma glucose and hemoglobin A1c were 81% (95% CI 76%-86%) and 70% (95% CI 67%-72%), and 80% (95% CI 66%-90%) and 69% (95% CI 58%-78%), respectively.ConclusionOur findings indicate that the 50 g GCT using the threshold of 140 mg/dL is a good screening test for identifying GDM at 24 to 28 weeks' gestational age for both high-risk and universal screening strategies in Southeast Asian countries. The non-fasting 2-hour PG, fasting plasma glucose or hemoglobin A1c are alternative choices for screening.
| S-EPMC7668444 | biostudies-literature