Dataset Information

Statistical uncertainty quantification to augment clinical decision support: a first implementation in sleep medicine.

ABSTRACT: Machine learning has the potential to change the practice of medicine, particularly in areas that require pattern recognition (e.g. radiology). Although automated classification is unlikely to be perfect, few modern machine learning tools have the ability to assess their own classification confidence to recognize uncertainty that might need human review. Using automated single-channel sleep staging as a first implementation, we demonstrated that uncertainty information (as quantified using Shannon entropy) can be utilized in a "human in the loop" methodology to promote targeted review of uncertain sleep stage classifications on an epoch-by-epoch basis. Across 20 sleep studies, this feedback methodology proved capable of improving scoring agreement with the gold standard over automated scoring alone (average improvement in Cohen's Kappa of 0.28), in a fraction of the scoring time compared to full manual review (60% reduction). In summary, our uncertainty-based clinician-in-the-loop framework promotes the improvement of medical classification accuracy/confidence in a cost-effective and economically resourceful manner.

SUBMITTER: Kang DY

PROVIDER: S-EPMC8484290 | biostudies-literature | 2021 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Statistical uncertainty quantification to augment clinical decision support: a first implementation in sleep medicine.

Kang Dae Y DY DeYoung Pamela N PN Tantiongloc Justin J Coleman Todd P TP Owens Robert L RL

NPJ digital medicine 20210930 1

Machine learning has the potential to change the practice of medicine, particularly in areas that require pattern recognition (e.g. radiology). Although automated classification is unlikely to be perfect, few modern machine learning tools have the ability to assess their own classification confidence to recognize uncertainty that might need human review. Using automated single-channel sleep staging as a first implementation, we demonstrated that uncertainty information (as quantified using Shann ...[more]

PMID: 34593972

Similar Datasets

Project description:BackgroundThe apnea-hypopnea index (AHI) (or one of its derivatives) is the primary clinical metric for characterizing sleep disordered breathing-the value of which with respect to a threshold determines severity of diagnosis and eligibility for treatment reimbursement. The index value, however, is taken as a perfect point estimate, with no measure of statistical uncertainty. Thus, current practice does not robustly account for variability in diagnosis/eligibility due to chance. In this paper, we quantify the statistical uncertainty associated with respiratory event indices for sleep disordered breathing and the effect of uncertainty on treatment eligibility.MethodsWe develop an empirical estimate of uncertainty using a non-parametric bootstrap on the interevent times, as well as a theoretical Poisson estimate reflecting the current formulation of the AHI. We then apply these methods to estimate AHI uncertainty for 2049 subjects (954/1095 M/F, age: mean 69 ± 9.1) from the Multi-Ethnic Study of Atherosclerosis (MESA).Results and conclusionsThe mean 95% empirical confidence interval width was 11.500 ± 6.208 events per hour and the mean 95% theoretical Poisson confidence interval width was 5.998 ± 2.897 events per hour, suggesting that uncertainty is likely a major confounding factor within the current diagnostic framework. Of the 278 subjects in the symptomatic population (ESS>10), 27% (76/278) had uncertain diagnoses given the 95% empirical confidence interval. Of the 2049 subjects in the full population, 43% (880/2049) had uncertain diagnoses given the 95% empirical confidence interval. The inclusion of subjects with uncertain diagnoses increases the number of eligible patients by 21.3% for the symptomatic population and by 84.8% for the full population. The exclusion of subjects with uncertain diagnoses given the 95% empirical confidence interval decreases the number of eligible patients by 12.4% for the symptomatic population and by 34.8% for full population. Additional analyses suggest that it is practically infeasible to gain diagnostic statistical significance through additional testing for a broad range of borderline cases. Overall, these results suggest that AHI uncertainty is a vital additional piece of information that would greatly benefit clinical practice, and that the inclusion of uncertainty in epidemiological analysis might help improve the ability for researchers to robustly link AHI with co-morbidities and long-term outcomes.

Project description:BackgroundThe emergency department is a critical juncture in the trajectory of care of patients with serious, life-limiting illness. Implementation of a clinical decision support (CDS) tool automates identification of older adults who may benefit from palliative care instead of relying upon providers to identify such patients, thus improving quality of care by assisting providers with adhering to guidelines. The Primary Palliative Care for Emergency Medicine (PRIM-ER) study aims to optimize the use of the electronic health record by creating a CDS tool to identify high risk patients most likely to benefit from primary palliative care and provide point-of-care clinical recommendations.MethodsA clinical decision support tool entitled Emergency Department Supportive Care Clinical Decision Support (Support-ED) was developed as part of an institutionally-sponsored value based medicine initiative at the Ronald O. Perelman Department of Emergency Medicine at NYU Langone Health. A multidisciplinary approach was used to develop Support-ED including: a scoping review of ED palliative care screening tools; launch of a workgroup to identify patient screening criteria and appropriate referral services; initial design and usability testing via the standard System Usability Scale questionnaire, education of the ED workforce on the Support-ED background, purpose and use, and; creation of a dashboard for monitoring and feedback.ResultsThe scoping review identified the Palliative Care and Rapid Emergency Screening (P-CaRES) survey as a validated instrument in which to adapt and apply for the creation of the CDS tool. The multidisciplinary workshops identified two primary objectives of the CDS: to identify patients with indicators of serious life limiting illness, and to assist with referrals to services such as palliative care or social work. Additionally, the iterative design process yielded three specific patient scenarios that trigger a clinical alert to fire, including: 1) when an advance care planning document was present, 2) when a patient had a previous disposition to hospice, and 3) when historical and/or current clinical data points identify a serious life-limiting illness without an advance care planning document present. Monitoring and feedback indicated a need for several modifications to improve CDS functionality.ConclusionsCDS can be an effective tool in the implementation of primary palliative care quality improvement best practices. Health systems should thoughtfully consider tailoring their CDSs in order to adapt to their unique workflows and environments. The findings of this research can assist health systems in effectively integrating a primary palliative care CDS system seamlessly into their processes of care.Trial registrationClinicalTrials.gov Identifier: NCT03424109. Registered 6 February 2018, Grant Number: AT009844-01.

Project description:BACKGROUND:Decisional tools have demonstrated their importance in informing manufacturing and commercial decisions in the monoclonal antibody domain. Recent approved therapies in regenerative medicine have shown great clinical benefits to patients. OBJECTIVE:The objective of this review was to investigate what decisional tools are available and what issues and gaps have been raised for their use in regenerative medicine. METHODS:We systematically searched MEDLINE to identify articles on decision support tools relevant to tissue engineering, and cell and gene therapy, with the aim of identifying gaps for future decisional tool development. We included published studies in English including a description of decisional tools in regenerative medicines. We extracted data using a predesigned Excel table and assessed the data both quantitatively and qualitatively. RESULTS:We identified 9 articles addressing key decisions in manufacturing and product development challenges in cell therapies. The decision objectives, parameters, assumptions, and solution methods were analyzed in detail. We found that all decisional tools focused on cell therapies, and 6 of the 9 reviews focused on allogeneic cell therapy products. We identified no available tools on tissue-engineering and gene therapy products. These studies addressed key decisions in manufacturing and product development challenges in cell therapies, such as choice of technology, through modeling. CONCLUSIONS:Our review identified a limited number of decisional tools. While the monoclonal antibodies and biologics decisional tool domain has been well developed and has shown great importance in driving more cost-effective manufacturing processes and better investment decisions, there is a lot to be learned in the regenerative medicine domain. There is ample space for expansion, especially with regard to autologous cell therapies, tissue engineering, and gene therapies. To consider the problem more comprehensively, the full needle-to-needle process should be modeled and evaluated.

Dataset Information

Statistical uncertainty quantification to augment clinical decision support: a first implementation in sleep medicine.

Publications

Statistical uncertainty quantification to augment clinical decision support: a first implementation in sleep medicine.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets