Dataset Information

Empirical assessment of ChatGPT's answering capabilities in natural science and engineering.

ABSTRACT: ChatGPT is a powerful language model from OpenAI that is arguably able to comprehend and generate text. ChatGPT is expected to greatly impact society, research, and education. An essential step to understand ChatGPT's expected impact is to study its domain-specific answering capabilities. Here, we perform a systematic empirical assessment of its abilities to answer questions across the natural science and engineering domains. We collected 594 questions on natural science and engineering topics from 198 faculty members across five faculties at Delft University of Technology. After collecting the answers from ChatGPT, the participants assessed the quality of the answers using a systematic scheme. Our results show that the answers from ChatGPT are, on average, perceived as "mostly correct". Two major trends are that the rating of the ChatGPT answers significantly decreases (i) as the educational level of the question increases and (ii) as we evaluate skills beyond scientific knowledge, e.g., critical attitude.

SUBMITTER: Schulze Balhorn L

PROVIDER: S-EPMC10904823 | biostudies-literature | 2024 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Empirical assessment of ChatGPT's answering capabilities in natural science and engineering.

Schulze Balhorn Lukas L Weber Jana M JM Buijsman Stefan S Hildebrandt Julian R JR Ziefle Martina M Schweidtmann Artur M AM

Scientific reports 20240229 1

ChatGPT is a powerful language model from OpenAI that is arguably able to comprehend and generate text. ChatGPT is expected to greatly impact society, research, and education. An essential step to understand ChatGPT's expected impact is to study its domain-specific answering capabilities. Here, we perform a systematic empirical assessment of its abilities to answer questions across the natural science and engineering domains. We collected 594 questions on natural science and engineering topics f ...[more]

PMID: 38424125

Similar Datasets

Project description:BackgroundGenerative artificial intelligence (AI) and large language models, such as OpenAI's ChatGPT, have shown promising potential in supporting medical education and clinical decision-making, given their vast knowledge base and natural language processing capabilities. As a general purpose AI system, ChatGPT can complete a wide range of tasks, including differential diagnosis without additional training. However, the specific application of ChatGPT in learning and applying a series of specialized, context-specific tasks mimicking the workflow of a human assessor, such as administering a standardized assessment questionnaire, followed by inputting assessment results in a standardized form, and interpretating assessment results strictly following credible, published scoring criteria, have not been thoroughly studied.ObjectiveThis exploratory study aims to evaluate and optimize ChatGPT's capabilities in administering and interpreting the Sour Seven Questionnaire, an informant-based delirium assessment tool. Specifically, the objectives were to train ChatGPT-3.5 and ChatGPT-4 to understand and correctly apply the Sour Seven Questionnaire to clinical vignettes using prompt engineering, assess the performance of these AI models in identifying and scoring delirium symptoms against scores from human experts, and refine and enhance the models' interpretation and reporting accuracy through iterative prompt optimization.MethodsWe used prompt engineering to train ChatGPT-3.5 and ChatGPT-4 models on the Sour Seven Questionnaire, a tool for assessing delirium through caregiver input. Prompt engineering is a methodology used to enhance the AI's processing of inputs by meticulously structuring the prompts to improve accuracy and consistency in outputs. In this study, prompt engineering involved creating specific, structured commands that guided the AI models in understanding and applying the assessment tool's criteria accurately to clinical vignettes. This approach also included designing prompts to explicitly instruct the AI on how to format its responses, ensuring they were consistent with clinical documentation standards.ResultsBoth ChatGPT models demonstrated promising proficiency in applying the Sour Seven Questionnaire to the vignettes, despite initial inconsistencies and errors. Performance notably improved through iterative prompt engineering, enhancing the models' capacity to detect delirium symptoms and assign scores. Prompt optimizations included adjusting the scoring methodology to accept only definitive "Yes" or "No" responses, revising the evaluation prompt to mandate responses in a tabular format, and guiding the models to adhere to the 2 recommended actions specified in the Sour Seven Questionnaire.ConclusionsOur findings provide preliminary evidence supporting the potential utility of AI models such as ChatGPT in administering standardized clinical assessment tools. The results highlight the significance of context-specific training and prompt engineering in harnessing the full potential of these AI models for health care applications. Despite the encouraging results, broader generalizability and further validation in real-world settings warrant additional research.

Project description:Introduction: Scientific communication, particularly the dissemination of research findings to both the scientific community and the general public, are skills required of graduates embarking on post-graduate studies and employment within the biomedical sciences sector. The aims of this action research project were to i) co-design an online scientific communication and digital capabilities resource, constructively aligned to the learning objectives of a final year undergraduate investigative research project; ii) ensure resource flexibility for future adaptation by others iii) embed authentic scientific communication learning assessments, namely, the preparation of a lay summary and visual abstract and iv) promote students' awareness of developed digital capabilities and transferable skills through written reflection. Materials and Methods: Student engagement, self-efficacy, experiences and performance and staff perceptions (n = 15) were evaluated by a mixed methods approach. Qualitative data was gathered from focus sessions, free text responses within questionnaires and content analysis of students' written reflections (n = 104). Quantitative data from 5-point Likert responses within student questionnaires (n = 31) and analysis of student scientific and lay writing (n = 146) using the readability parameters Flesch-Kincaid Grade Level and Flesch Reading Ease were analysed using non-parametric statistical methods. Results: A learning resource was co-designed with students, staff, local, national and international contributors and valued by both students and staff, enabling students to prepare scientific communication outputs of a professional standard by application of digital, analytical and scientific communication skills. Students prepared lay summaries which were statistically (p < 0.0001) more readable than their paired scientific abstracts. Significant correlations between easier readability of lay summaries and awarded marks for the written elements of the module were noted. Students reported their digital and communication capabilities increased significantly (p < 0.0001) throughout, from limited to good/excellent and reflected on the numerous transferable skills developed during preparation of assessments, with 75% reflecting on their digital capabilities. Discussion: Undergraduate students developed, appreciated and used varied scientific communication and digital skills to articulate research findings. The embedding of such activities throughout all levels of higher education will enable students to develop their digital and scientific skills and reflect on the development of such transferable skills for application in their future careers.

Project description:BackgroundCervical cancer (CC) and breast cancer (BC) threaten women's well-being, influenced by health-related stigma and a lack of reliable information, which can cause late diagnosis and early death. ChatGPT is likely to become a key source of health information, although quality concerns could also influence health-seeking behaviours.MethodsThis cross-sectional online survey compared ChatGPT's responses to five physicians specializing in mammography and five specializing in gynaecology. Twenty frequently asked questions about CC and BC were asked on 26th and 29th of April, 2023. A panel of seven experts assessed the accuracy, consistency, and relevance of ChatGPT's responses using a 7-point Likert scale. Responses were analyzed for readability, reliability, and efficiency. ChatGPT's responses were synthesized, and findings are presented as a radar chart.ResultsChatGPT had an accuracy score of 7.0 (range: 6.6-7.0) for CC and BC questions, surpassing the highest-scoring physicians (P < 0.05). ChatGPT took an average of 13.6 s (range: 7.6-24.0) to answer each of the 20 questions presented. Readability was comparable to that of experts and physicians involved, but ChatGPT generated more extended responses compared to physicians. The consistency of repeated answers was 5.2 (range: 3.4-6.7). With different contexts combined, the overall ChatGPT relevance score was 6.5 (range: 4.8-7.0). Radar plot analysis indicated comparably good accuracy, efficiency, and to a certain extent, relevance. However, there were apparent inconsistencies, and the reliability and readability be considered inadequate.ConclusionsChatGPT shows promise as an initial source of information for CC and BC. ChatGPT is also highly functional and appears to be superior to physicians, and aligns with expert consensus, although there is room for improvement in readability, reliability, and consistency. Future efforts should focus on developing advanced ChatGPT models explicitly designed to improve medical practice and for those with concerns about symptoms.

Dataset Information

Empirical assessment of ChatGPT's answering capabilities in natural science and engineering.

Publications

Empirical assessment of ChatGPT's answering capabilities in natural science and engineering.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets