Dataset Information

The Impact of Information Relevancy and Interactivity on Intensivists' Trust in a Machine Learning-Based Bacteremia Prediction System: Simulation Study.

ABSTRACT:

Background

The exponential growth in computing power and the increasing digitization of information have substantially advanced the machine learning (ML) research field. However, ML algorithms are often considered "black boxes," and this fosters distrust. In medical domains, in which mistakes can result in fatal outcomes, practitioners may be especially reluctant to trust ML algorithms.

Objective

The aim of this study is to explore the effect of user-interface design features on intensivists' trust in an ML-based clinical decision support system.

Methods

A total of 47 physicians from critical care specialties were presented with 3 patient cases of bacteremia in the setting of an ML-based simulation system. Three conditions of the simulation were tested according to combinations of information relevancy and interactivity. Participants' trust in the system was assessed by their agreement with the system's prediction and a postexperiment questionnaire. Linear regression models were applied to measure the effects.

Results

Participants' agreement with the system's prediction did not differ according to the experimental conditions. However, in the postexperiment questionnaire, higher information relevancy ratings and interactivity ratings were associated with higher perceived trust in the system (P<.001 for both). The explicit visual presentation of the features of the ML algorithm on the user interface resulted in lower trust among the participants (P=.05).

Conclusions

Information relevancy and interactivity features should be considered in the design of the user interface of ML-based clinical decision support systems to enhance intensivists' trust. This study sheds light on the connection between information relevancy, interactivity, and trust in human-ML interaction, specifically in the intensive care unit environment.

SUBMITTER: Katzburg O

PROVIDER: S-EPMC11310737 | biostudies-literature | 2024 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The Impact of Information Relevancy and Interactivity on Intensivists' Trust in a Machine Learning-Based Bacteremia Prediction System: Simulation Study.

Katzburg Omer O Roimi Michael M Frenkel Amit A Ilan Roy R Bitan Yuval Y

JMIR human factors 20240801

<h4>Background</h4>The exponential growth in computing power and the increasing digitization of information have substantially advanced the machine learning (ML) research field. However, ML algorithms are often considered "black boxes," and this fosters distrust. In medical domains, in which mistakes can result in fatal outcomes, practitioners may be especially reluctant to trust ML algorithms.<h4>Objective</h4>The aim of this study is to explore the effect of user-interface design features on i ...[more]

PMID: 39092520

Similar Datasets

Project description:IntroductionBacteremia is a common but life-threatening infectious disease. However, a well-defined rule to assess patient risk of bacteremia and the urgency of blood culture is lacking. The aim of this study is to establish a predictive model for bacteremia in septic patients using available big data in the emergency department (ED) through logistic regression and other machine learning (ML) methods.Material and methodsWe conducted a retrospective cohort study at the ED of National Cheng Kung University Hospital in Taiwan from January 2015 to December 2019. ED adults (≥18 years old) with systemic inflammatory response syndrome and receiving blood cultures during the ED stay were included. Models I and II were established based on logistic regression, both of which were derived from support vector machine (SVM) and random forest (RF). Net reclassification index was used to determine which model was superior.ResultsDuring the study period, 437,969 patients visited the study ED, and 40,395 patients were enrolled. Patients diagnosed with bacteremia accounted for 7.7% of the cohort. The area under the receiver operating curve (AUROC) in models I and II was 0.729 (95% CI, 0.718-0.740) and 0.731 (95% CI, 0.721-0.742), with Akaike information criterion (AIC) of 16,840 and 16,803, respectively. The performance of model II was superior to that of model I. The AUROC values of models III and IV in the validation dataset were 0.730 (95% CI, 0.713-0.747) and 0.705 (0.688-0.722), respectively. There is no statistical evidence to support that the performance of the model created with logistic regression is superior to those created by SVM and RF.DiscussionThe advantage of the SVM or RF model is that the prediction model is more elastic and not limited to a linear relationship. The advantage of the LR model is that it is easy to explain the influence of the independent variable on the response variable. These models could help medical staff identify high-risk patients and prevent unnecessary antibiotic use. The performance of SVM and RF was not inferior to that of logistic regression.ConclusionsWe established models that provide discrimination in predicting bacteremia among patients with sepsis. The reported results could inspire researchers to adopt ML in their development of prediction algorithms.

Project description:Chemical synaptic transmission involves the release of a neurotransmitter that diffuses in the extracellular space and interacts with specific receptors located on the postsynaptic membrane. Computer simulation approaches provide fundamental tools for exploring various aspects of the synaptic transmission under different conditions. In particular, Monte Carlo methods can track the stochastic movements of neurotransmitter molecules and their interactions with other discrete molecules, the receptors. However, these methods are computationally expensive, even when used with simplified models, preventing their use in large-scale and multi-scale simulations of complex neuronal systems that may involve large numbers of synaptic connections. We have developed a machine-learning based method that can accurately predict relevant aspects of the behavior of synapses, such as the percentage of open synaptic receptors as a function of time since the release of the neurotransmitter, with considerably lower computational cost compared with the conventional Monte Carlo alternative. The method is designed to learn patterns and general principles from a corpus of previously generated Monte Carlo simulations of synapses covering a wide range of structural and functional characteristics. These patterns are later used as a predictive model of the behavior of synapses under different conditions without the need for additional computationally expensive Monte Carlo simulations. This is performed in five stages: data sampling, fold creation, machine learning, validation and curve fitting. The resulting procedure is accurate, automatic, and it is general enough to predict synapse behavior under experimental conditions that are different to the ones it has been trained on. Since our method efficiently reproduces the results that can be obtained with Monte Carlo simulations at a considerably lower computational cost, it is suitable for the simulation of high numbers of synapses and it is therefore an excellent tool for multi-scale simulations.

Dataset Information

The Impact of Information Relevancy and Interactivity on Intensivists' Trust in a Machine Learning-Based Bacteremia Prediction System: Simulation Study.

Background

Objective

Methods

Results

Conclusions

Publications

The Impact of Information Relevancy and Interactivity on Intensivists' Trust in a Machine Learning-Based Bacteremia Prediction System: Simulation Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets