Dataset Information

A large-scale dataset of patient summaries for retrieval-based clinical decision support systems.

ABSTRACT: Retrieval-based Clinical Decision Support (ReCDS) can aid clinical workflow by providing relevant literature and similar patients for a given patient. However, the development of ReCDS systems has been severely obstructed by the lack of diverse patient collections and publicly available large-scale patient-level annotation datasets. In this paper, we collect a novel dataset of patient summaries and relations called PMC-Patients to benchmark two ReCDS tasks: Patient-to-Article Retrieval (ReCDS-PAR) and Patient-to-Patient Retrieval (ReCDS-PPR). Specifically, we extract patient summaries from PubMed Central articles using simple heuristics and utilize the PubMed citation graph to define patient-article relevance and patient-patient similarity. PMC-Patients contains 167k patient summaries with 3.1 M patient-article relevance annotations and 293k patient-patient similarity annotations, which is the largest-scale resource for ReCDS and also one of the largest patient collections. Human evaluation and analysis show that PMC-Patients is a diverse dataset with high-quality annotations. We also implement and evaluate several ReCDS systems on the PMC-Patients benchmarks to show its challenges and conduct several case studies to show the clinical utility of PMC-Patients.

SUBMITTER: Zhao Z

PROVIDER: S-EPMC10728216 | biostudies-literature | 2023 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A large-scale dataset of patient summaries for retrieval-based clinical decision support systems.

Zhao Zhengyun Z Jin Qiao Q Chen Fangyuan F Peng Tuorui T Yu Sheng S

Scientific data 20231218 1

Retrieval-based Clinical Decision Support (ReCDS) can aid clinical workflow by providing relevant literature and similar patients for a given patient. However, the development of ReCDS systems has been severely obstructed by the lack of diverse patient collections and publicly available large-scale patient-level annotation datasets. In this paper, we collect a novel dataset of patient summaries and relations called PMC-Patients to benchmark two ReCDS tasks: Patient-to-Article Retrieval (ReCDS-PA ...[more]

PMID: 38110415

Similar Datasets

Project description:ObjectiveTo develop and validate a questionnaire to assess the usability of clinical decision support systems (CDSS) and to assist in the early identification of usability issues that may impact patient safety and quality of care.DesignMixed research methods were used to develop and validate the questionnaire. The qualitative study involved scale item development, content and face validity. Pilot testing established construct validity using factor analysis and facilitated estimates for reliability and internal consistency using the Cronbach's alpha coefficient.SettingTwo hospitals within a single National Health Service Trust.ParticipantsWe recruited a panel of 7 experts in usability and questionnaire writing for health purposes to test content validity; 10 participants to assess face validity and 78 participants for the pilot testing. To be eligible for this last phase, participants needed to be health professionals with at least 3 months experience using the local hospital electronic patient record system.ResultsFeedback from the face and content validity phases contributed to the development and improvement of scale items. The final Healthcare Systems Usability Scale (HSUS) proved quick to complete, easy to understand and was mostly worded by potential users. Exploratory analysis revealed four factors related to patient safety, task execution, alerts or recommendations accuracy, the effects of the system on workflow and ease of system use. These separate into four subscales: patient safety and decision effectiveness (seven items), workflow integration (six items), work effectiveness (five items) and user control (four items). These factors affect the quality of care and clinician's ability to make informed and timely decisions when using CDSS. The HSUS has a very good reliability with global Cronbach's alpha 0.914 and between 0.702 and 0.926 for the four subscales.ConclusionThe HSUS is a valid and reliable tool for usability testing of CDSS and early identification of usability issues that may cause medical adverse events.

Project description:BackgroundClinical Decision Support Systems (CDSS) are used to assist the decision-making process in the healthcare field. Developing an effective CDSS is an arduous task that can take advantage from prior assessment of the most promising theories, techniques and methods used at the present time.ObjectiveTo identify the features of Clinical Decision Support Systems and provide an analysis of their effectiveness. Thus, two research questions were formulated: RQ1-What are the most common trend characteristics in a CDSS? RQ2-What is the maturity level of the CDSS based on the decision-making theory proposed by Simon?MethodsAIS e-library, Decision Support Systems journal, Nature, PlosOne and PubMed were selected as information sources to conduct this systematic literature review. Studies from 2000 to 2020 were chosen covering search terms in CDSS, selected according to defined eligibility criteria. The data were extracted and managed in a worksheet, based on the defined criteria. PRISMA statements were used to report the systematic review.ResultsThe outcomes showed that rule-based module was the most used approach regarding knowledge management and representation. The most common technological feature adopted by the CDSS were the recommendations and suggestions. 19,23% of studies adopt the type of system as a web-based application, and 51,92% are standalone CDSS. Temporal evolution was also possible to visualize. This study contributed to the development of a Maturity Staging Model, where it was possible to verify that most CDSS do not exceed level 2 of maturity.ConclusionThe trend characteristics addressed in the revised CDSS were identified, compared to the four predefined groups. A maturity stage model was developed based on Simon's decision-making theory, allowing to assess the level of maturity of the most common features of the CDSS. With the application of the model, it was noticed that the phases of choice and implementation are underrepresented. This constitutes the main gap in the development of an effective CDSS.

Project description:BackgroundComputerized clinical decision support systems (CDSSs) are a promising knowledge translation tool, but often fail to meaningfully influence the outcomes they target. Low CDSS provider uptake is a potential contributor to this problem but has not been systematically studied. The objective of this systematic review and meta-regression was to determine reported CDSS uptake and identify which CDSS features may influence uptake.MethodsMedline, Embase, CINAHL, and the Cochrane Database of Controlled Trials were searched from January 2000 to August 2020. Randomized, non-randomized, and quasi-experimental trials reporting CDSS uptake in any patient population or setting were included. The main outcome extracted was CDSS uptake, reported as a raw proportion, and representing the number of times the CDSS was used or accessed over the total number of times it could have been interacted with. We also extracted context, content, system, and implementation features that might influence uptake, for each CDSS. Overall weighted uptake was calculated using random-effects meta-analysis and determinants of uptake were investigated using multivariable meta-regression.ResultsAmong 7995 citations screened, 55 studies involving 373,608 patients and 3607 providers met full inclusion criteria. Meta-analysis revealed that overall CDSS uptake was 34.2% (95% CI 23.2 to 47.1%). Uptake was only reported in 12.4% of studies that otherwise met inclusion criteria. Multivariable meta-regression revealed the following factors significantly associated with uptake: (1) formally evaluating the availability and quality of the patient data needed to inform CDSS advice; and (2) identifying and addressing other barriers to the behaviour change targeted by the CDSS.Conclusions and relevanceSystem uptake was seldom reported in CDSS trials. When reported, uptake was low. This represents a major and potentially modifiable barrier to overall CDSS effectiveness. We found that features relating to CDSS context and implementation strategy best predicted uptake. Future studies should measure the impact of addressing these features as part of the CDSS implementation strategy. Uptake reporting must also become standard in future studies reporting CDSS intervention effects.RegistrationPre-registered on PROSPERO, CRD42018092337.

Project description:BackgroundDiagnostic errors pose significant health risks and contribute to patient mortality. With the growing accessibility of electronic health records, machine learning models offer a promising avenue for enhancing diagnosis quality. Current research has primarily focused on a limited set of diseases with ample training data, neglecting diagnostic scenarios with limited data availability.ObjectiveThis study aims to develop an information retrieval (IR)-based framework that accommodates data sparsity to facilitate broader diagnostic decision support.MethodsWe introduced an IR-based diagnostic decision support framework called CliniqIR. It uses clinical text records, the Unified Medical Language System Metathesaurus, and 33 million PubMed abstracts to classify a broad spectrum of diagnoses independent of training data availability. CliniqIR is designed to be compatible with any IR framework. Therefore, we implemented it using both dense and sparse retrieval approaches. We compared CliniqIR's performance to that of pretrained clinical transformer models such as Clinical Bidirectional Encoder Representations from Transformers (ClinicalBERT) in supervised and zero-shot settings. Subsequently, we combined the strength of supervised fine-tuned ClinicalBERT and CliniqIR to build an ensemble framework that delivers state-of-the-art diagnostic predictions.ResultsOn a complex diagnosis data set (DC3) without any training data, CliniqIR models returned the correct diagnosis within their top 3 predictions. On the Medical Information Mart for Intensive Care III data set, CliniqIR models surpassed ClinicalBERT in predicting diagnoses with <5 training samples by an average difference in mean reciprocal rank of 0.10. In a zero-shot setting where models received no disease-specific training, CliniqIR still outperformed the pretrained transformer models with a greater mean reciprocal rank of at least 0.10. Furthermore, in most conditions, our ensemble framework surpassed the performance of its individual components, demonstrating its enhanced ability to make precise diagnostic predictions.ConclusionsOur experiments highlight the importance of IR in leveraging unstructured knowledge resources to identify infrequently encountered diagnoses. In addition, our ensemble framework benefits from combining the complementary strengths of the supervised and retrieval-based models to diagnose a broad spectrum of diseases.

Dataset Information

A large-scale dataset of patient summaries for retrieval-based clinical decision support systems.

Publications

A large-scale dataset of patient summaries for retrieval-based clinical decision support systems.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets