Dataset Information

Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system.

ABSTRACT:

Background

For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required.

Objective

The aim of this review study was to form a practical framework and identify key design factors for experimental design in evaluating the performance of clinicians with or without the aid of ML-CDSS.

Methods

The study was based on published ML-CDSS performance evaluation studies. We systematically searched articles published between January 2016 and December 2022. From the articles we collected a set of design factors. Only the articles comparing the performance of clinicians with or without the aid of ML-CDSS using experimental study methods were considered.

Results

The identified key design factors for the practical framework of ML-CDSS experimental study design were performance measures, user interface, ground truth data and the selection of samples and participants. In addition, we identified the importance of randomization, crossover design and training and practice rounds. Previous studies had shortcomings in the rationale and documentation of choices regarding the number of participants and the duration of the experiment.

Conclusion

The design factors of ML-CDSS experimental study are interdependent and all factors must be considered in individual choices.

Supplementary information

The online version contains supplementary material available at 10.1007/s12553-023-00763-1.

SUBMITTER: Nuutinen M

PROVIDER: S-EPMC10262137 | biostudies-literature | 2023 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system.

Nuutinen Mikko M Leskelä Riikka-Leena RL

Health and technology 20230613

<h4>Background</h4>For the adoption of machine learning clinical decision support systems (ML-CDSS) it is critical to understand the performance aid of the ML-CDSS. However, it is not trivial, how the performance aid should be evaluated. To design reliable performance evaluation study, both the knowledge from the practical framework of experimental study design and the understanding of domain specific design factors are required.<h4>Objective</h4>The aim of this review study was to form a practi ...[more]

PMID: 37363342

Similar Datasets

Project description:IntroductionSpirometry is a point-of-care lung function test that helps support the diagnosis and monitoring of chronic lung disease. The quality and interpretation accuracy of spirometry is variable in primary care. This study aims to evaluate whether artificial intelligence (AI) decision support software improves the performance of primary care clinicians in the interpretation of spirometry, against reference standard (expert interpretation).Methods and analysisA parallel, two-group, statistician-blinded, randomised controlled trial of primary care clinicians in the UK, who refer for, or interpret, spirometry. People with specialist training in respiratory medicine to consultant level were excluded. A minimum target of 228 primary care clinician participants will be randomised with a 1:1 allocation to assess fifty de-identified, real-world patient spirometry sessions through an online platform either with (intervention group) or without (control group) AI decision support software report. Outcomes will cover primary care clinicians' spirometry interpretation performance including measures of technical quality assessment, spirometry pattern recognition and diagnostic prediction, compared with reference standard. Clinicians' self-rated confidence in spirometry interpretation will also be evaluated. The primary outcome is the proportion of the 50 spirometry sessions where the participant's preferred diagnosis matches the reference diagnosis. Unpaired t-tests and analysis of covariance will be used to estimate the difference in primary outcome between intervention and control groups.Ethics and disseminationThis study has been reviewed and given favourable opinion by Health Research Authority Wales (reference: 22/HRA/5023). Results will be submitted for publication in peer-reviewed journals, presented at relevant national and international conferences, disseminated through social media, patient and public routes and directly shared with stakeholders.Trial registration numberNCT05933694.

Project description:ImportanceAn increasing number of machine learning (ML)-based clinical decision support systems (CDSSs) are described in the medical literature, but this research focuses almost entirely on comparing CDSS directly with clinicians (human vs computer). Little is known about the outcomes of these systems when used as adjuncts to human decision-making (human vs human with computer).ObjectivesTo conduct a systematic review to investigate the association between the interactive use of ML-based diagnostic CDSSs and clinician performance and to examine the extent of the CDSSs' human factors evaluation.Evidence reviewA search of MEDLINE, Embase, PsycINFO, and grey literature was conducted for the period between January 1, 2010, and May 31, 2019. Peer-reviewed studies published in English comparing human clinician performance with and without interactive use of an ML-based diagnostic CDSSs were included. All metrics used to assess human performance were considered as outcomes. The risk of bias was assessed using Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) and Risk of Bias in Non-Randomised Studies-Intervention (ROBINS-I). Narrative summaries were produced for the main outcomes. Given the heterogeneity of medical conditions, outcomes of interest, and evaluation metrics, no meta-analysis was performed.FindingsA total of 8112 studies were initially retrieved and 5154 abstracts were screened; of these, 37 studies met the inclusion criteria. The median number of participating clinicians was 4 (interquartile range, 3-8). Of the 107 results that reported statistical significance, 54 (50%) were increased by the use of CDSSs, 4 (4%) were decreased, and 49 (46%) showed no change or an unclear change. In the subgroup of studies carried out in representative clinical settings, no association between the use of ML-based diagnostic CDSSs and improved clinician performance could be observed. Interobserver agreement was the commonly reported outcome whose change was the most strongly associated with CDSS use. Four studies (11%) reported on user feedback, and, in all but 1 case, clinicians decided to override at least some of the algorithms' recommendations. Twenty-eight studies (76%) were rated as having a high risk of bias in at least 1 of the 4 QUADAS-2 core domains, and 6 studies (16%) were considered to be at serious or critical risk of bias using ROBINS-I.Conclusions and relevanceThis systematic review found only sparse evidence that the use of ML-based CDSSs is associated with improved clinician diagnostic performance. Most studies had a low number of participants, were at high or unclear risk of bias, and showed little or no consideration for human factors. Caution should be exercised when estimating the current potential of ML to improve human diagnostic performance, and more comprehensive evaluation should be conducted before deploying ML-based CDSSs in clinical settings. The results highlight the importance of considering supported human decisions as end points rather than merely the stand-alone CDSSs outputs.

Dataset Information

Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system.

Background

Objective

Methods

Results

Conclusion

Supplementary information

Publications

Systematic review of the performance evaluation of clinicians with or without the aid of machine learning clinical decision support system.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets