Question answering systems for health professionals at the point of care-a systematic review.
Ontology highlight
ABSTRACT: Objectives
Question answering (QA) systems have the potential to improve the quality of clinical care by providing health professionals with the latest and most relevant evidence. However, QA systems have not been widely adopted. This systematic review aims to characterize current medical QA systems, assess their suitability for healthcare, and identify areas of improvement.Materials and methods
We searched PubMed, IEEE Xplore, ACM Digital Library, ACL Anthology, and forward and backward citations on February 7, 2023. We included peer-reviewed journal and conference papers describing the design and evaluation of biomedical QA systems. Two reviewers screened titles, abstracts, and full-text articles. We conducted a narrative synthesis and risk of bias assessment for each study. We assessed the utility of biomedical QA systems.Results
We included 79 studies and identified themes, including question realism, answer reliability, answer utility, clinical specialism, systems, usability, and evaluation methods. Clinicians' questions used to train and evaluate QA systems were restricted to certain sources, types and complexity levels. No system communicated confidence levels in the answers or sources. Many studies suffered from high risks of bias and applicability concerns. Only 8 studies completely satisfied any criterion for clinical utility, and only 7 reported user evaluations. Most systems were built with limited input from clinicians.Discussion
While machine learning methods have led to increased accuracy, most studies imperfectly reflected real-world healthcare information needs. Key research priorities include developing more realistic healthcare QA datasets and considering the reliability of answer sources, rather than merely focusing on accuracy.
SUBMITTER: Kell G
PROVIDER: S-EPMC10990539 | biostudies-literature | 2024 Apr
REPOSITORIES: biostudies-literature
Publications
Question answering systems for health professionals at the point of care-a systematic review.
Kell Gregory G Roberts Angus A Umansky Serge S Qian Linglong L Ferrari Davide D Soboczenski Frank F Wallace Byron C BC Patel Nikhil N Marshall Iain J IJ
Journal of the American Medical Informatics Association : JAMIA 20240401 4
<h4>Objectives</h4>Question answering (QA) systems have the potential to improve the quality of clinical care by providing health professionals with the latest and most relevant evidence. However, QA systems have not been widely adopted. This systematic review aims to characterize current medical QA systems, assess their suitability for healthcare, and identify areas of improvement.<h4>Materials and methods</h4>We searched PubMed, IEEE Xplore, ACM Digital Library, ACL Anthology, and forward and ...[more]
Similar Datasets
Project description:BackgroundQuestion answering (QA) systems for patient-related data can assist both clinicians and patients. They can, for example, assist clinicians in decision-making and enable patients to have a better understanding of their medical history. Substantial amounts of patient data are stored in electronic health records (EHRs), making EHR QA an important research area. Because of the differences in data format and modality, this differs greatly from other medical QA tasks that use medical websites or scientific papers to retrieve answers, making it critical to research EHR QA.ObjectiveThis study aims to provide a methodological review of existing works on QA for EHRs. The objectives of this study were to identify the existing EHR QA datasets and analyze them, study the state-of-the-art methodologies used in this task, compare the different evaluation metrics used by these state-of-the-art models, and finally elicit the various challenges and the ongoing issues in EHR QA.MethodsWe searched for articles from January 1, 2005, to September 30, 2023, in 4 digital sources, including Google Scholar, ACL Anthology, ACM Digital Library, and PubMed, to collect relevant publications on EHR QA. Our systematic screening process followed PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. A total of 4111 papers were identified for our study, and after screening based on our inclusion criteria, we obtained 47 papers for further study. The selected studies were then classified into 2 non-mutually exclusive categories depending on their scope: "EHR QA datasets" and "EHR QA models."ResultsA systematic screening process obtained 47 papers on EHR QA for final review. Out of the 47 papers, 53% (n=25) were about EHR QA datasets, and 79% (n=37) papers were about EHR QA models. It was observed that QA on EHRs is relatively new and unexplored. Most of the works are fairly recent. In addition, it was observed that emrQA is by far the most popular EHR QA dataset, both in terms of citations and usage in other papers. We have classified the EHR QA datasets based on their modality, and we have inferred that Medical Information Mart for Intensive Care (MIMIC-III) and the National Natural Language Processing Clinical Challenges datasets (ie, n2c2 datasets) are the most popular EHR databases and corpuses used in EHR QA. Furthermore, we identified the different models used in EHR QA along with the evaluation metrics used for these models.ConclusionsEHR QA research faces multiple challenges, such as the limited availability of clinical annotations, concept normalization in EHR QA, and challenges faced in generating realistic EHR QA datasets. There are still many gaps in research that motivate further work. This study will assist future researchers in focusing on areas of EHR QA that have possible future research directions.
| S-EPMC11561445 | biostudies-literature
Project description:BACKGROUND:User-friendly information at the point of care should be well structured, rapidly accessible, and comprehensive. Also, this information should be trustworthy, as it will be used by health care practitioners to practice evidence-based medicine. Therefore, a standard, validated tool to evaluate the trustworthiness of such point-of-care information resources is needed. OBJECTIVE:This systematic review sought to search for tools to assess the trustworthiness of point-of-care resources and to describe and analyze the content of these tools. METHODS:A systematic search was performed on three sources: (1) we searched online for initiatives that worked off of the trustworthiness of medical information; (2) we searched Medline (PubMed) until June 2019 for relevant literature; and (3) we scanned reference lists and lists of citing papers via Web of Science for each retrieved paper. We included all studies, reports, websites, or methodologies that reported on tools that assessed the trustworthiness of medical information for professionals. From the selected studies, we extracted information on the general characteristics of the tools. As no standard, risk-of-bias assessment instruments are available for these types of studies, we described how each tool was developed, including any assessments on reliability and validity. We analyzed the criteria used in the different tools and divided them into five categories: (1) author-related information; (2) evidence-based methodology; (3) website quality; (4) website design and usability; and (5) website interactivity. The percentage of tools in compliance with these categories and the different criteria were calculated. RESULTS:Included in this review was a total of 17 tools, all published between 1997 and 2018. The tools were developed for different purposes, from a general quality assessment of medical information to very detailed analyses, all specifically for point-of-care resources. However, the development process of the tools was poorly described. Overall, seven tools had a scoring system implemented, two were assessed for reliability only, and two other tools were assessed for both validity and reliability. The content analysis showed that all the tools assessed criteria related to an evidence-based methodology: 82% of the tools assessed author-related information, 71% assessed criteria related to website quality, 71% assessed criteria related to website design and usability, and 47% of the tools assessed criteria related to website interactivity. There was significant variability in criteria used, as some were very detailed while others were more broadly defined. CONCLUSIONS:The 17 included tools encompass a variety of items important for the assessment of the trustworthiness of point-of-care information. Overall, two tools were assessed for both reliability and validity, but they lacked some essential criteria for the assessment of the trustworthiness of medical information for use at the point-of-care. Currently, a standard, validated tool does not exist. The results of this review may contribute to the development of such an instrument, which may enhance the quality of point-of-care information in the long term. TRIAL REGISTRATION:PROSPERO CRD42019122565; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=122565.
| S-EPMC6996752 | biostudies-literature
Project description:BackgroundDigitalization is not fully implemented in clinical practice, and several factors have been identified as possible barriers, including the competencies of health care professionals. However, no summary of the available evidence has been provided to date to depict digital health competencies that have been investigated among health care professionals, the tools used in assessing such competencies, and the effective interventions to improve them.ObjectiveThis review aims to summarize digital health competencies investigated to date and the tools used to assess them among health care professionals.MethodsA systematic review based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist was performed. The MEDLINE, Cumulative Index to Nursing and Allied Health Literature, PsycINFO, and Scopus databases were accessed up to September 4, 2021. Studies assessing digital health competencies with quantitative designs, targeting health care professionals, and written in English were included. The methodological quality of included studies was evaluated using the Joanna Briggs Institute tools.ResultsA total of 26 studies, published from 1999 to 2021, met the inclusion criteria, and the majority were cross sectional in design, while only 2 were experimental study designs. Most studies were assessed with moderate to low methodological quality; 4 categories and 9 subcategories of investigated digital health competencies have been identified. The most investigated category was "Self-rated competencies," followed by "Psychological and emotional aspects toward digital technologies," "Use of digital technologies," and "Knowledge about digital technologies." In 35% (9/26) of the studies, a previously validated tool was used to measure the competencies assessed, while others developed ad hoc questionnaires.ConclusionsMainly descriptive studies with issues regarding methodology quality have been produced to date investigating 4 main categories of digital health competencies mostly with nonvalidated tools. Competencies investigated might be considered while designing curricula for undergraduate, postgraduate, and continuing education processes, whereas the methodological lacks detected might be addressed with future research. There is a need to expand research on psychological and emotional elements and the ability to use digital technology to self-learn and teach others.Trial registrationPROSPERO International Prospective Register of Systematic Reviews CRD42021282775; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=282775.
| S-EPMC9437781 | biostudies-literature
Project description:BackgroundThe digital transformation of healthcare enables new ways of working in cancer care directing attention on the digital skills of healthcare professionals. This systematic review aims to identify existing evidence about digital skills among health care professionals in cancer care to identify the needs for future education and research.MethodsDatabase searches were conducted in PubMed, CINAHL, Web of Science, Scopus, Cochrane and ERIC to identify studies until March 2023. The inclusion criteria were digital skills of health care professionals in cancer care as described by themselves, other health care professionals, patients or significant others. The CASP tool was used for quality assessment of the studies. Data was analysed following inductive content analysis.ResultsThe search produced 4563 records, of which 24 studies were included (12 qualitative, 10 quantitative, 1 mixed methods design and 1 strategy paper). Four main categories were identified describing HCPs' required skills, existing skills and development areas of digital skills in cancer care: Skills for information technology, Skills for ethical practice, Skills for creating a human-oriented relationship and Skills for digital education and support. In development areas, one more main category, Skills for implementing digital health, was identified.ConclusionThe digital skills of health care professionals in cancer care are multifaceted and fundamental for quality cancer care. The skills need to be assessed to provide education based on actual learning needs. The review findings can be used for education and research in this field.
| S-EPMC10962045 | biostudies-literature
Project description:Biomedical professionals have access to a huge amount of literature, but when they use a search engine, they often have to deal with too many documents to efficiently find the appropriate information in a reasonable time. In this perspective, question-answering (QA) engines are designed to display answers, which were automatically extracted from the retrieved documents. Standard QA engines in literature process a user question, then retrieve relevant documents and finally extract some possible answers out of these documents using various named-entity recognition processes. In our study, we try to answer complex genomics questions, which can be adequately answered only using Gene Ontology (GO) concepts. Such complex answers cannot be found using state-of-the-art dictionary- and redundancy-based QA engines. We compare the effectiveness of two dictionary-based classifiers for extracting correct GO answers from a large set of 100 retrieved abstracts per question. In the same way, we also investigate the power of GOCat, a GO supervised classifier. GOCat exploits the GOA database to propose GO concepts that were annotated by curators for similar abstracts. This approach is called deep QA, as it adds an original classification step, and exploits curated biological data to infer answers, which are not explicitly mentioned in the retrieved documents. We show that for complex answers such as protein functional descriptions, the redundancy phenomenon has a limited effect. Similarly usual dictionary-based approaches are relatively ineffective. In contrast, we demonstrate how existing curated data, beyond information extraction, can be exploited by a supervised classifier, such as GOCat, to massively improve both the quantity and the quality of the answers with a +100% improvement for both recall and precision. Database URL: http://eagl.unige.ch/DeepQA4PA/.
| S-EPMC4572360 | biostudies-literature
Project description:BackgroundThe proliferation of the scientific literature in the field of biomedicine makes it difficult to keep abreast of current knowledge, even for domain experts. While general Web search engines and specialized information retrieval (IR) systems have made important strides in recent decades, the problem of accurate knowledge extraction from the biomedical literature is far from solved. Classical IR systems usually return a list of documents that have to be read by the user to extract relevant information. This tedious and time-consuming work can be lessened with automatic Question Answering (QA) systems, which aim to provide users with direct and precise answers to their questions. In this work we propose a novel methodology for QA based on semantic relations extracted from the biomedical literature.ResultsWe extracted semantic relations with the SemRep natural language processing system from 122,421,765 sentences, which came from 21,014,382 MEDLINE citations (i.e., the complete MEDLINE distribution up to the end of 2012). A total of 58,879,300 semantic relation instances were extracted and organized in a relational database. The QA process is implemented as a search in this database, which is accessed through a Web-based application, called SemBT (available at http://sembt.mf.uni-lj.si ). We conducted an extensive evaluation of the proposed methodology in order to estimate the accuracy of extracting a particular semantic relation from a particular sentence. Evaluation was performed by 80 domain experts. In total 7,510 semantic relation instances belonging to 2,675 distinct relations were evaluated 12,083 times. The instances were evaluated as correct 8,228 times (68%).ConclusionsIn this work we propose an innovative methodology for biomedical QA. The system is implemented as a Web-based application that is able to provide precise answers to a wide range of questions. A typical question is answered within a few seconds. The tool has some extensions that make it especially useful for interpretation of DNA microarray results.
| S-EPMC4307891 | biostudies-literature
Project description:BACKGROUND: Cost consequences analysis was completed from randomized controlled trial (RCT) data for the Just-in-time (JIT) librarian consultation service in primary care that ran from October 2005 to April 2006. The service was aimed at providing answers to clinical questions arising during the clinical encounter while the patient waits. Cost saving and cost avoidance were also analyzed. The data comes from eighty-eight primary care providers in the Ottawa area working in Family Health Networks (FHNs) and Family Health Groups (FHGs). METHODS: We conducted a cost consequences analysis based on data from the JIT project. We also estimated the potential economic benefit of JIT librarian consultation service to the health care system. RESULTS: The results show that the cost per question for the JIT service was $38.20. The cost could be as low as $5.70 per question for a regular service. Nationally, if this service was implemented and if family physicians saw additional patients when the JIT service saved them time, up to 61,100 extra patients could be seen annually. A conservative estimate of the cost savings and cost avoidance per question for JIT was $11.55. CONCLUSIONS: The cost per question, if the librarian service was used at full capacity, is quite low. Financial savings to the health care system might exceed the cost of the service. Saving physician's time during their day could potentially lead to better access to family physicians by patients. Implementing a librarian consultation service can happen quickly as the time required to train professional librarians to do this service is short.
| S-EPMC3307768 | biostudies-other
Project description:MotivationCurrent studies in extractive question answering (EQA) have modeled the single-span extraction setting, where a single answer span is a label to predict for a given question-passage pair. This setting is natural for general domain EQA as the majority of the questions in the general domain can be answered with a single span. Following general domain EQA models, current biomedical EQA (BioEQA) models utilize the single-span extraction setting with post-processing steps.ResultsIn this paper, we investigate the question distribution across the general and biomedical domains and discover biomedical questions are more likely to require list-type answers (multiple answers) than factoid-type answers (single answer). This necessitates the models capable of producing multiple answers for a question. Based on this preliminary study, we propose a sequence tagging approach for BioEQA, which is a multi-span extraction setting. Our approach directly tackles questions with a variable number of phrases as their answer and can learn to decide the number of answers for a question from training data. Our experimental results on the BioASQ 7 b and 8 b list-type questions outperformed the best-performing existing models without requiring post-processing steps.AvailabilitySource codes and resources are freely available for download at https://github.com/dmis-lab/SeqTagQA.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC9344839 | biostudies-literature
Project description:The free-energy principle (FEP) is a formal model of neuronal processes that is widely recognised in neuroscience as a unifying theory of the brain and biobehaviour. More recently, however, it has been extended beyond the brain to explain the dynamics of living systems, and their unique capacity to avoid decay. The aim of this review is to synthesise these advances with a meta-theoretical ontology of biological systems called variational neuroethology, which integrates the FEP with Tinbergen's four research questions to explain biological systems across spatial and temporal scales. We exemplify this framework by applying it to Homo sapiens, before translating variational neuroethology into a systematic research heuristic that supplies the biological, cognitive, and social sciences with a computationally tractable guide to discovery.
| S-EPMC5857288 | biostudies-literature
Project description:ObjectivePoint-of-care testing (POCT) has become an essential diagnostic technology for optimal patient care. Its implementation, however, still falls behind. This paper reviews the available evidence on the health economic impact of introducing POCT to assess if poor POCT uptake may be related to lacking evidence.Study designThe Scopus and PubMed databases were searched to identify publications describing a health economic evaluation of a point-of-care (POC) test. Data were extracted from the included publications, including general and methodological characteristics as well as the study results summarized in either cost, effects or an incremental cost-effectiveness ratio. Results were sorted into six groups according to the POC test's purpose (diagnosis, screening or monitoring) and care setting (primary care or secondary care). The reporting quality of the publications was determined using the CHEERS checklist.ResultsThe initial search resulted in 396 publications, of which 44 met the inclusion criteria. Most of the evaluations were performed in a primary care setting (n = 31; 70.5%) compared with a secondary care setting (n = 13; 29.5%). About two thirds of the evaluations were on POC tests implemented with a diagnostic purpose (n = 28; 63.6%). More than 75% of evaluations concluded that POCT is recommended for implementation, although in some cases only under specific circumstances and conditions. Compliance with the CHEERS checklist items ranged from 20.8% to 100%, with an average reporting quality of 72.0%.ConclusionThere were very few evaluations in this review that advised against the implementation of POCT. However, the uptake of POCT in many countries remains low. Even though the evaluations included in this review did not always include the full long-term benefits of POCT, it is clear that health economic evidence across a few dimensions of value already indicate the benefits of POCT. This suggests that the lack of evidence on POCT is not the primary barrier to its implementation and that the low uptake of these tests in clinical practice is due to (a combination of) other barriers. In this context, aspects around organization of care, support of clinicians and quality management may be crucial in the widespread implementation of POCT.
| S-EPMC8160040 | biostudies-literature