Sources of bias in diagnostic accuracy studies and the diagnostic process.
Ontology highlight
ABSTRACT:
SUBMITTER: Furukawa TA
PROVIDER: S-EPMC1373753 | biostudies-literature | 2006 Feb
REPOSITORIES: biostudies-literature
Publications
Sources of bias in diagnostic accuracy studies and the diagnostic process.
Furukawa Toshi A TA Guyatt Gordon H GH
CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne 20060201 4
Similar Datasets
Project description:ObjectiveMachine learning (ML) diagnostic tools have significant potential to improve health care. However, methodological pitfalls may affect diagnostic test accuracy studies used to appraise such tools. We aimed to evaluate the prevalence and reporting of design characteristics within the literature. Further, we sought to empirically assess whether design features may be associated with different estimates of diagnostic accuracy.Materials and methodsWe systematically retrieved 2 × 2 tables (n = 281) describing the performance of ML diagnostic tools, derived from 114 publications in 38 meta-analyses, from PubMed. Data extracted included test performance, sample sizes, and design features. A mixed-effects metaregression was run to quantify the association between design features and diagnostic accuracy.ResultsParticipant ethnicity and blinding in test interpretation was unreported in 90% and 60% of studies, respectively. Reporting was occasionally lacking for rudimentary characteristics such as study design (28% unreported). Internal validation without appropriate safeguards was used in 44% of studies. Several design features were associated with larger estimates of accuracy, including having unreported (relative diagnostic odds ratio [RDOR], 2.11; 95% confidence interval [CI], 1.43-3.1) or case-control study designs (RDOR, 1.27; 95% CI, 0.97-1.66), and recruiting participants for the index test (RDOR, 1.67; 95% CI, 1.08-2.59).DiscussionSignificant underreporting of experimental details was present. Study design features may affect estimates of diagnostic performance in the ML diagnostic test accuracy literature.ConclusionsThe present study identifies pitfalls that threaten the validity, generalizability, and clinical value of ML diagnostic tools and provides recommendations for improvement.
| S-EPMC7647361 | biostudies-literature
Project description:BackgroundObstetrics and gynaecology have seen rapid growth in the development of new tests with research on these tests presented as diagnostic accuracy studies. To avoid errors in judgement it is important that the methodology of these studies is such that bias is minimised. Our objective was to determine the methodological quality of test accuracy studies in obstetrics and gynaecology using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) checklist and to assess sources of bias.MethodsA prospective protocol was developed to assess the impact of QUADAS on ten systematic reviews performed over the period 2004-2007.We investigated whether there was an improvement in study quality since the introduction of QUADAS, whether a correlation existed between study sample size, country of origin of study and its quality. We also investigated whether there was a correlation between reporting and methodological quality and by the use of meta-regression analyses explored for items of quality that were associated with bias.ResultsA total of 300 studies were included. The overall quality of included studies was poor (> 50% compliance with 57.1% of quality items). However, the mean compliance with QUADAS showed an improvement post-publication of QUADAS (54.9% versus 61.4% p = 0.002). There was no correlation with study sample size. Gynaecology studies published from the United States of America showed higher quality (USA versus Western Europe p = 0.002; USA versus Asia p = 0.004). Meta-regression analysis showed that no individual quality item had a significant impact on accuracy. There was an association between reporting and methodological quality (r = 0.51 p < 0.0001 for obstetrics and r = 0.56 p < 0.0001 for gynaecology).ConclusionsA combination of poor methodological quality and poor reporting affects the inferences that can be drawn from test accuracy studies. Further compliance with quality checklists is required to ensure that bias is minimised.
| S-EPMC3072918 | biostudies-literature
Project description:BackgroundPoor reporting quality in diagnostic accuracy studies hampers an adequate judgment of the validity of the study. The Standards for Reporting of Diagnostic Accuracy Studies (STARD) statement was published to improve the reporting quality of diagnostic accuracy studies. This study aimed to evaluate the adherence of diagnostic accuracy studies published in Annals of Laboratory Medicine (ALM) to STARD 2015 and to identify directions for improvement in the reporting quality of these studies.MethodsTwo independent authors assessed articles published in ALM between 2012-2018 for compliance with 30 STARD 2015 checklist items to identify all eligible diagnostic accuracy studies published during this period. We included 66 diagnostic accuracy studies. A total of the fulfilled STARD items were calculated, and adherence was analyzed on an individual-item basis.ResultsThe overall mean±SD number of STARD items reported for the included studies was 11.2±2.7. Only five (7.6%) studies adhered to more than 50% of the 30 items. No study satisfied more than 80% of the items. Large variability in adherence to reporting standards was detected across items, ranging from 0% to 100%.ConclusionsAdherence to STARD 2015 is suboptimal among diagnostic accuracy studies published in ALM. Our study emphasizes the necessity of adherence to STARD to improve the reporting quality of future diagnostic accuracy studies to be published in ALM.
| S-EPMC6933069 | biostudies-literature
Project description:IntroductionCognitive screening measures often lack sensitivity and are hampered by inequities across ethnoracial groups. A multitrait multimethod (MTMM) classification may attenuate these shortcomings.MethodsA sample of 7227 participants across the diagnostic spectrum were selected from the National Alzheimer's Coordinating Center cohort. Random forest ensemble methods were used to predict diagnosis across the sample and within Black American (n = 1025) and non-Hispanic White groups (n = 5263) based on: (1) a demographically corrected Montreal Cognitive Assessment (MoCA), (2) MoCA and Functional Assessment Questionnaire (FAQ), (3) MoCA and FAQ with demographic correction.ResultsThe MTMM approach with demographic correction had the highest diagnostic accuracy for the cognitively unimpaired (area under curve [AUC] [95% confidence interval (CI)]): 0.906 [0.892, 0.920]) and mild cognitive impairment (AUC: 0.835 [0.810, 0.860]) groups and reduced racial disparities.DiscussionWith further validation, the MTMM approach combining cognitive screening and functional status assessment may serve to improve diagnostic accuracy and extend opportunities for early intervention with greater equity.
| S-EPMC8652409 | biostudies-literature
Project description:The development of a new diagnostic test ideally follows a sequence of stages which, among other aims, evaluate technical performance. This includes an analytical validity study, a diagnostic accuracy study, and an interventional clinical utility study. In this article, we propose a novel Bayesian approach to sample size determination for the diagnostic accuracy study, which takes advantage of information available from the analytical validity stage. We utilize assurance to calculate the required sample size based on the target width of a posterior probability interval and can choose to use or disregard the data from the analytical validity study when subsequently inferring measures of test accuracy. Sensitivity analyses are performed to assess the robustness of the proposed sample size to the choice of prior, and prior-data conflict is evaluated by comparing the data to the prior predictive distributions. We illustrate the proposed approach using a motivating real-life application involving a diagnostic test for ventilator associated pneumonia. Finally, we compare the properties of the approach against commonly used alternatives. The results show that, when suitable prior information is available, the assurance-based approach can reduce the required sample size when compared to alternative approaches.
| S-EPMC9325402 | biostudies-literature
Project description:BackgroundMetacognition is a cognitive process that involves self-awareness of thinking, understanding, and performance. This study assesses pathologists' metacognition by examining the association between their diagnostic accuracy and self-reported confidence levels while interpreting skin and breast biopsies.DesignWe studied 187 pathologists from the Melanoma Pathology Study (M-Path) and 115 pathologists from the Breast Pathology Study (B-Path). We measured pathologists' metacognitive ability by examining the area under the curve (AUC), the area under each pathologist's receiver operating characteristic (ROC) curve summarizing the association between confidence and diagnostic accuracy. We investigated possible relationships between this AUC measure, referred to as metacognitive sensitivity, and pathologist attributes. We also assessed whether higher metacognitive sensitivity affected the association between diagnostic accuracy and a secondary diagnostic action such as requesting a second opinion.ResultsWe found no significant associations between pathologist clinical attributes and metacognitive AUC. However, we found that pathologists with higher AUC showed a stronger trend to request secondary diagnostic action for inaccurate diagnoses and not for accurate diagnoses compared with pathologists with lower AUC.LimitationsPathologists reported confidence in specific diagnostic terms, rather than the broader classes into which the diagnostic terms were later grouped to determine accuracy. In addition, while there is no gold standard for the correct diagnosis to determine the accuracy of pathologists' interpretations, our studies achieved a high-quality reference diagnosis by using the consensus diagnosis of 3 experienced pathologists.ConclusionsMetacognition can affect clinical decisions. If pathologists have self-awareness that their diagnosis may be inaccurate, they can request additional tests or second opinions, providing the opportunity to correct inaccurate diagnoses.HighlightsMetacognitive sensitivity varied across pathologists, with most showing higher sensitivity than expected by chance.None of the demographic or clinical characteristics we examined was significantly associated with metacognitive sensitivity.Pathologists with higher metacognitive sensitivity were more likely to request additional tests or second opinions for their inaccurate diagnoses.
| S-EPMC9825636 | biostudies-literature
Project description:The number of diagnostic test accuracy (DTA) studies concerning biomarkers have gradually increased during the past years. However, study designs remain imperfect, and the statistical methods used are not meaningful in some published studies. Here, we introduce recommendations for designing DTA studies, including consecutive enrollment of participants with uniform inclusion and exclusion criteria, blinded testing and interpretation, prespecified thresholds, and the use of one reference standard for all subjects. In addition, we also describe more relevant statistical methods in DTA studies, including decision curve analysis (DCA), nomograms, diagnostic model and scale, net reclassification index (NRI), and the integrated discriminatory index (IDI). This review may help clinicians to better design DTA studies that investigating biomarkers.
| S-EPMC6989996 | biostudies-literature
Project description:Caries diagnostic studies differ with respect to their design, included patients/tooth samples, use of diagnostic and reference methods, calibration, blinding and data reporting. Such heterogeneity makes comparisons between studies difficult and could represent a substantial risk of bias (RoB) when it is not identified. Therefore, the present report aims to describe the development and background of a RoB assessment tool for caries diagnostic studies. The expert group developed and agreed to use a RoB assessment tool during three workshops. Here, existing instruments (e.g., QUADAS 2 and the Joanna Briggs Institute Reviewers' Manual) influenced the hierarchy and phrasing of the signalling questions that were adapted to the specific dental purpose. The tailored RoB assessment tool that was created consists of 16 signalling questions that are organized in four domains. This tool considers the selection/spectrum bias (1), the bias of the index (2) and reference tests (3), and the bias of the study flow and data analysis (4) and can be downloaded from the journal website. This paper explores possible sources of heterogeneity and bias in caries diagnostic studies and summarizes the relevant methodological aspects.
| S-EPMC8347423 | biostudies-literature
Project description:IntroductionThe Kaiser Healthy Aging and Diverse Life Experiences (KHANDLE) study enrolled Asian, Black, Latino, and White adults ages 65+ without prior dementia diagnosis (N = 1709). We evaluated the prevalence of cognitive impairment (mild cognitive impairment or dementia) accounting for potential biases.MethodsA random subgroup (N = 541) received clinical evaluation and others were evaluated if they failed a cognitive screen. Diagnoses were made under two conditions: (1) demographics-blind, based on clinical exam and demographically adjusted neuropsychological test scores; and (2) all available information (clinical exam, demographics, and adjusted and unadjusted test scores).ResultsCognitive impairment prevalence was 28% for blinded-adjusted diagnosis and 25% using all available information. Black participants had higher impairment rates than White (both conditions) and Latino (blinded-adjusted diagnosis) participants. Incomplete assessments negatively biased prevalence estimates for White participants.DiscussionRacial/ethnic disparities in cognitive impairment were amplified by attrition bias in White participants but were unaffected by type of test norms and diagnosticians' knowledge of demographics.
| S-EPMC8719430 | biostudies-literature
Project description:INTRODUCTION. Liquid biopsies are a minimally invasive collection of a patient body fluid sample. In oncology, they offer several advantages compared to traditional tissue biopsies. However, potential of this method in endometrial cancer (EC) remains poorly explored. We studied the utility of tumor educated platelets (TEPs) and circulating tumor DNA (ctDNA) for preoperative EC diagnosis, including histology determination. MATERIALS AND METHODS. TEPs from 297 subjects (53 EC patients, 40 patients with benign gynecologic conditions and 204 healthy women) were RNA-sequenced. DNA sequencing was performed in 519 primary tumor tissue samples and in16 plasma samples. Artificial intelligence was applied to sample classification. RESULTS. Platelet-dedicated classifier yielded AUC of 93.1% in test set when discriminating between healthy subjects and cancer patients. However, the discrimination between endometrial cancer and benign gynecologic conditions was relatively low, with AUC of 60.7%. ctDNA-dedicated classifier discriminated primary tumor tissue samples with AUC of 91.4% and ctDNA blood samples with AUC of 87.5%. CONCLUSIONS Liquid biopsies show potential in EC diagnosis. Both TEPs and ctDNA profiles coupled with artificial intelligence constitute a source of useful information. Further work, involving more cases, is warranted.
2021-12-01 | GSE184904 | GEO