Dataset Information

Empirical data drift detection experiments on real-world medical imaging data.

ABSTRACT: While it is common to monitor deployed clinical artificial intelligence (AI) models for performance degradation, it is less common for the input data to be monitored for data drift - systemic changes to input distributions. However, when real-time evaluation may not be practical (eg., labeling costs) or when gold-labels are automatically generated, we argue that tracking data drift becomes a vital addition for AI deployments. In this work, we perform empirical experiments on real-world medical imaging to evaluate three data drift detection methods' ability to detect data drift caused (a) naturally (emergence of COVID-19 in X-rays) and (b) synthetically. We find that monitoring performance alone is not a good proxy for detecting data drift and that drift-detection heavily depends on sample size and patient features. Our work discusses the need and utility of data drift detection in various scenarios and highlights gaps in knowledge for the practical application of existing methods.

SUBMITTER: Kore A

PROVIDER: S-EPMC10904813 | biostudies-literature | 2024 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Empirical data drift detection experiments on real-world medical imaging data.

Kore Ali A Abbasi Bavil Elyar E Subasri Vallijah V Abdalla Moustafa M Fine Benjamin B Dolatabadi Elham E Abdalla Mohamed M

Nature communications 20240229 1

While it is common to monitor deployed clinical artificial intelligence (AI) models for performance degradation, it is less common for the input data to be monitored for data drift - systemic changes to input distributions. However, when real-time evaluation may not be practical (eg., labeling costs) or when gold-labels are automatically generated, we argue that tracking data drift becomes a vital addition for AI deployments. In this work, we perform empirical experiments on real-world medical i ...[more]

PMID: 38424096

Similar Datasets

Project description:BackgroundElectronic medical records (EMRs) have the highest value among real-world data (RWD). The aim of the present study was to propose a data collection framework of EMR-based RWD to evaluate the effectiveness and safety of cancer drugs by conducting a nationwide real-world study based on the Korean Cancer Study Group.MethodsWe considered all patients who received ramucirumab plus paclitaxel (RAM/PTX) for gastric cancer and trastuzumab emtansine (T-DM1) for breast cancer at relevant institutions in South Korea. Standard operating procedures for systematic data collection were prospectively developed. Investigator reliability was evaluated using the concordance rate between the recommended input value for representative fictional cases and the input value of each investigator. Reliability of collected data was evaluated twice during the study period at three institutions randomly selected using the concordance rate between the previously collected data and data collected by an independent investigator. The reliability results of the investigators and collected data were used for revision of the electronic data capture system and site training.ResultsBetween the starting date of medical insurance coverage and December 2018, a total of 1063 patients at 56 institutions in the RAM/PTX cohort and 824 patients at 60 institutions in the T-DM1 cohort were included. Mean investigator reliability in the RAM/PTX and T-DM1 cohorts was 73.5% and 71.9%, respectively. Mean reliability of collected data in the RAM/PTX and T-DM1 cohort was 90.0% for both cohorts in the first analysis and 89.0% and 84.0% in the second analysis, respectively. Mean missing values of the RAM/PTX and T-DM1 cohorts at the time of simulation of fictional cases and final data analysis decreased from 20.7% to 0.46% and from 18.5% to 0.76%, respectively.ConclusionThis real-world study provides a framework that ensures relevance and reliability of EMR-based RWD for evaluating the effectiveness and safety of cancer drugs.

Project description:BackgroundAdverse events (AEs) associated with vaccination have traditionally been evaluated by epidemiological studies. More recently, they have gained attention due to the emergency use authorization of several COVID-19 vaccines. As part of its responsibility to conduct postmarket surveillance, the US Food and Drug Administration continues to monitor several AEs of interest to ensure the safety of vaccines, including those for COVID-19.ObjectiveThis study is part of the Biologics Effectiveness and Safety Initiative, which aims to improve the US Food and Drug Administration's postmarket surveillance capabilities while minimizing the burden of collecting clinical data on suspected postvaccination AEs. The objective of this study was to enhance active surveillance efforts through a pilot platform that can receive automatically reported AE cases through a health care data exchange.MethodsWe detected cases by sharing and applying computable phenotype algorithms to real-world data in health care providers' electronic health records databases. Using the fast healthcare interoperability resources standard for secure data transmission, we implemented a computable phenotype algorithm on a new health care system. The study focused on the algorithm's positive predictive value, validated through clinical records, assessing both the time required for implementation and the accuracy of AE detection.ResultsThe algorithm required 200-250 hours to implement and optimize. Of the 6,574,420 clinical encounters across 694,151 patients, 30 cases were identified as potential myocarditis/pericarditis. Of these, 26 cases were retrievable, and 24 underwent clinical validation. In total, 14 cases were confirmed as definite or probable myocarditis/pericarditis, yielding a positive predictive value of 58.3% (95% CI 37.3%-76.9%). These findings underscore the algorithm's capability for real-time detection of AEs, though they also highlight variability in performance across different health care systems.ConclusionsThe study advocates for the ongoing refinement and application of distributed computable phenotype algorithms to enhance AE detection capabilities. These tools are crucial for comprehensive postmarket surveillance and improved vaccine safety monitoring. The outcomes suggest the need for further optimization to achieve more consistent results across diverse health care settings.

Dataset Information

Empirical data drift detection experiments on real-world medical imaging data.

Publications

Empirical data drift detection experiments on real-world medical imaging data.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets