Dataset Information

Needs Assessment Survey Identifying Research Processes Which may be Improved by Automation or Artificial Intelligence: ICU Community Modeling and Artificial Intelligence to Improve Efficiency (ICU-Comma).

ABSTRACT:

Background

Critical care research in Canada is conducted primarily in academically-affiliated intensive care units with established research infrastructure, including research coordinators (RCs). Recently, efforts have been made to engage community hospital ICUs in research albeit with barriers. Automation or artificial intelligence (AI) could aid the performance of routine research tasks. It is unclear which research study processes might be improved through AI automation.

Methods

We conducted a cross-sectional survey of Canadian ICU research personnel. The survey contained items characterizing opinions regarding research processes that may be amenable to AI automation. We distributed the questionnaire via email distribution lists of 3 Canadian research societies. Open-ended questions were analyzed using a thematic content analysis approach.

Results

A total of 49 survey responses were received (response rate: 8%). Tasks that respondents felt were time-consuming/tedious/tiresome included: screening for potentially eligible patients (74%), inputting data into case report forms (65%), and preparing internal tracking logs (53%). Tasks that respondents felt could be performed by AI automation included: screening for eligible patients (59%), inputting data into case report forms (55%), preparing internal tracking logs (51%), and randomizing patients into studies (45%). Open-ended questions identified enthusiasm for AI automation to improve information accuracy and efficiency while freeing up RCs to perform tasks that require human interaction. This enthusiasm was tempered by the need for proper AI education and oversight.

Conclusions

There were balanced supportive (increased efficiency and re-allocation of tasks) and challenges (informational accuracy and oversight) with regards to AI automation in ICU research.

SUBMITTER: Lau VI

PROVIDER: S-EPMC9468938 | biostudies-literature | 2022 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Needs Assessment Survey Identifying Research Processes Which may be Improved by Automation or Artificial Intelligence: ICU Community Modeling and Artificial Intelligence to Improve Efficiency (ICU-Comma).

Lau Vincent I VI Binnie Alexandra A Basmaji John J Baig Nadia N Opgenorth Dawn D Cameron Saoirse S O'Hearn Katie K McDonald Ellen E Senaratne Janek J Sligl Wendy W Zuege Danny J DJ Rewa Oleksa O Bagshaw Sean M SM Tsang Jennifer J

Journal of intensive care medicine 20211213 10

<h4>Background</h4>Critical care research in Canada is conducted primarily in academically-affiliated intensive care units with established research infrastructure, including research coordinators (RCs). Recently, efforts have been made to engage community hospital ICUs in research albeit with barriers. Automation or artificial intelligence (AI) could aid the performance of routine research tasks. It is unclear which research study processes might be improved through AI automation.<h4>Methods</h ...[more]

PMID: 34898324

Similar Datasets

Project description:AimThe aim of this study was to evaluate and compare artificial intelligence (AI)-based large language models (LLMs) (ChatGPT-3.5, Bing, and Bard) with human-based formulations in generating relevant clinical queries, using comprehensive methodological evaluations.MethodsTo interact with the major LLMs ChatGPT-3.5, Bing Chat, and Google Bard, scripts and prompts were designed to formulate PICOT (population, intervention, comparison, outcome, time) clinical questions and search strategies. Quality of the LLMs responses was assessed using a descriptive approach and independent assessment by two researchers. To determine the number of hits, PubMed, Web of Science, Cochrane Library, and CINAHL Ultimate search results were imported separately, without search restrictions, with the search strings generated by the three LLMs and an additional one by the expert. Hits from one of the scenarios were also exported for relevance evaluation. The use of a single scenario was chosen to provide a focused analysis. Cronbach's alpha and intraclass correlation coefficient (ICC) were also calculated.ResultsIn five different scenarios, ChatGPT-3.5 generated 11,859 hits, Bing 1,376,854, Bard 16,583, and an expert 5919 hits. We then used the first scenario to assess the relevance of the obtained results. The human expert search approach resulted in 65.22% (56/105) relevant articles. Bing was the most accurate AI-based LLM with 70.79% (63/89), followed by ChatGPT-3.5 with 21.05% (12/45), and Bard with 13.29% (42/316) relevant hits. Based on the assessment of two evaluators, ChatGPT-3.5 received the highest score (M = 48.50; SD = 0.71). Results showed a high level of agreement between the two evaluators. Although ChatGPT-3.5 showed a lower percentage of relevant hits compared to Bing, this reflects the nuanced evaluation criteria, where the subjective evaluation prioritized contextual accuracy and quality over mere relevance.ConclusionThis study provides valuable insights into the ability of LLMs to formulate PICOT clinical questions and search strategies. AI-based LLMs, such as ChatGPT-3.5, demonstrate significant potential for augmenting clinical workflows, improving clinical query development, and supporting search strategies. However, the findings also highlight limitations that necessitate further refinement and continued human oversight.Clinical relevanceAI could assist nurses in formulating PICOT clinical questions and search strategies. AI-based LLMs offer valuable support to healthcare professionals by improving the structure of clinical questions and enhancing search strategies, thereby significantly increasing the efficiency of information retrieval.

Dataset Information

Needs Assessment Survey Identifying Research Processes Which may be Improved by Automation or Artificial Intelligence: ICU Community Modeling and Artificial Intelligence to Improve Efficiency (ICU-Comma).

Background

Methods

Results

Conclusions

Publications

Needs Assessment Survey Identifying Research Processes Which may be Improved by Automation or Artificial Intelligence: ICU Community Modeling and Artificial Intelligence to Improve Efficiency (ICU-Comma).

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets