Dataset Information

Perception, performance, and detectability of conversational artificial intelligence across 32 university courses.

ABSTRACT: The emergence of large language models has led to the development of powerful tools such as ChatGPT that can produce text indistinguishable from human-generated work. With the increasing accessibility of such technology, students across the globe may utilize it to help with their school work-a possibility that has sparked ample discussion on the integrity of student evaluation processes in the age of artificial intelligence (AI). To date, it is unclear how such tools perform compared to students on university-level courses across various disciplines. Further, students' perspectives regarding the use of such tools in school work, and educators' perspectives on treating their use as plagiarism, remain unknown. Here, we compare the performance of the state-of-the-art tool, ChatGPT, against that of students on 32 university-level courses. We also assess the degree to which its use can be detected by two classifiers designed specifically for this purpose. Additionally, we conduct a global survey across five countries, as well as a more in-depth survey at the authors' institution, to discern students' and educators' perceptions of ChatGPT's use in school work. We find that ChatGPT's performance is comparable, if not superior, to that of students in a multitude of courses. Moreover, current AI-text classifiers cannot reliably detect ChatGPT's use in school work, due to both their propensity to classify human-written answers as AI-generated, as well as the relative ease with which AI-generated text can be edited to evade detection. Finally, there seems to be an emerging consensus among students to use the tool, and among educators to treat its use as plagiarism. Our findings offer insights that could guide policy discussions addressing the integration of artificial intelligence into educational frameworks.

SUBMITTER: Ibrahim H

PROVIDER: S-EPMC10449897 | biostudies-literature | 2023 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Perception, performance, and detectability of conversational artificial intelligence across 32 university courses.

Ibrahim Hazem H Liu Fengyuan F Asim Rohail R Battu Balaraju B Benabderrahmane Sidahmed S Alhafni Bashar B Adnan Wifag W Alhanai Tuka T AlShebli Bedoor B Baghdadi Riyadh R Bélanger Jocelyn J JJ Beretta Elena E Celik Kemal K Chaqfeh Moumena M Daqaq Mohammed F MF Bernoussi Zaynab El ZE Fougnie Daryl D Garcia de Soto Borja B Gandolfi Alberto A Gyorgy Andras A Habash Nizar N Harris J Andrew JA Kaufman Aaron A Kirousis Lefteris L Kocak Korhan K Lee Kangsan K Lee Seungah S SS Malik Samreen S Maniatakos Michail M Melcher David D Mourad Azzam A Park Minsu M Rasras Mahmoud M Reuben Alicja A Zantout Dania D Gleason Nancy W NW Makovi Kinga K Rahwan Talal T Zaki Yasir Y

Scientific reports 20230824 1

The emergence of large language models has led to the development of powerful tools such as ChatGPT that can produce text indistinguishable from human-generated work. With the increasing accessibility of such technology, students across the globe may utilize it to help with their school work-a possibility that has sparked ample discussion on the integrity of student evaluation processes in the age of artificial intelligence (AI). To date, it is unclear how such tools perform compared to students ...[more]

PMID: 37620342

Similar Datasets

Project description:BackgroundA rising number of conversational agents or chatbots are equipped with artificial intelligence (AI) architecture. They are increasingly prevalent in health care applications such as those providing education and support to patients with chronic diseases, one of the leading causes of death in the 21st century. AI-based chatbots enable more effective and frequent interactions with such patients.ObjectiveThe goal of this systematic literature review is to review the characteristics, health care conditions, and AI architectures of AI-based conversational agents designed specifically for chronic diseases.MethodsWe conducted a systematic literature review using PubMed MEDLINE, EMBASE, PyscInfo, CINAHL, ACM Digital Library, ScienceDirect, and Web of Science. We applied a predefined search strategy using the terms "conversational agent," "healthcare," "artificial intelligence," and their synonyms. We updated the search results using Google alerts, and screened reference lists for other relevant articles. We included primary research studies that involved the prevention, treatment, or rehabilitation of chronic diseases, involved a conversational agent, and included any kind of AI architecture. Two independent reviewers conducted screening and data extraction, and Cohen kappa was used to measure interrater agreement.A narrative approach was applied for data synthesis.ResultsThe literature search found 2052 articles, out of which 10 papers met the inclusion criteria. The small number of identified studies together with the prevalence of quasi-experimental studies (n=7) and prevailing prototype nature of the chatbots (n=7) revealed the immaturity of the field. The reported chatbots addressed a broad variety of chronic diseases (n=6), showcasing a tendency to develop specialized conversational agents for individual chronic conditions. However, there lacks comparison of these chatbots within and between chronic diseases. In addition, the reported evaluation measures were not standardized, and the addressed health goals showed a large range. Together, these study characteristics complicated comparability and open room for future research. While natural language processing represented the most used AI technique (n=7) and the majority of conversational agents allowed for multimodal interaction (n=6), the identified studies demonstrated broad heterogeneity, lack of depth of reported AI techniques and systems, and inconsistent usage of taxonomy of the underlying AI software, further aggravating comparability and generalizability of study results.ConclusionsThe literature on AI-based conversational agents for chronic conditions is scarce and mostly consists of quasi-experimental studies with chatbots in prototype stage that use natural language processing and allow for multimodal user interaction. Future research could profit from evidence-based evaluation of the AI-based conversational agents and comparison thereof within and between different chronic health conditions. Besides increased comparability, the quality of chatbots developed for specific chronic conditions and their subsequent impact on the target patients could be enhanced by more structured development and standardized evaluation processes.

Project description:The integration of artificial intelligence (AI) into health communication systems has introduced a transformative approach to public health management, particularly during public health emergencies, capable of reaching billions through familiar digital channels. This paper explores the utility and implications of generalist conversational artificial intelligence (CAI) advanced AI systems trained on extensive datasets to handle a wide range of conversational tasks across various domains with human-like responsiveness. The specific focus is on the application of generalist CAI within messaging services, emphasizing its potential to enhance public health communication. We highlight the evolution and current applications of AI-driven messaging services, including their ability to provide personalized, scalable, and accessible health interventions. Specifically, we discuss the integration of large language models and generative AI in mainstream messaging platforms, which potentially outperform traditional information retrieval systems in public health contexts. We report a critical examination of the advantages of generalist CAI in delivering health information, with a case of its operationalization during the COVID-19 pandemic and propose the strategic deployment of these technologies in collaboration with public health agencies. In addition, we address significant challenges and ethical considerations, such as AI biases, misinformation, privacy concerns, and the required regulatory oversight. We envision a future with leverages generalist CAI in messaging apps, proposing a multiagent approach to enhance the reliability and specificity of health communications. We hope this commentary initiates the necessary conversations and research toward building evaluation approaches, adaptive strategies, and robust legal and technical frameworks to fully realize the benefits of AI-enhanced communications in public health, aiming to ensure equitable and effective health outcomes across diverse populations.

Project description:BackgroundThe high demand for health care services and the growing capability of artificial intelligence have led to the development of conversational agents designed to support a variety of health-related activities, including behavior change, treatment support, health monitoring, training, triage, and screening support. Automation of these tasks could free clinicians to focus on more complex work and increase the accessibility to health care services for the public. An overarching assessment of the acceptability, usability, and effectiveness of these agents in health care is needed to collate the evidence so that future development can target areas for improvement and potential for sustainable adoption.ObjectiveThis systematic review aims to assess the effectiveness and usability of conversational agents in health care and identify the elements that users like and dislike to inform future research and development of these agents.MethodsPubMed, Medline (Ovid), EMBASE (Excerpta Medica dataBASE), CINAHL (Cumulative Index to Nursing and Allied Health Literature), Web of Science, and the Association for Computing Machinery Digital Library were systematically searched for articles published since 2008 that evaluated unconstrained natural language processing conversational agents used in health care. EndNote (version X9, Clarivate Analytics) reference management software was used for initial screening, and full-text screening was conducted by 1 reviewer. Data were extracted, and the risk of bias was assessed by one reviewer and validated by another.ResultsA total of 31 studies were selected and included a variety of conversational agents, including 14 chatbots (2 of which were voice chatbots), 6 embodied conversational agents (3 of which were interactive voice response calls, virtual patients, and speech recognition screening systems), 1 contextual question-answering agent, and 1 voice recognition triage system. Overall, the evidence reported was mostly positive or mixed. Usability and satisfaction performed well (27/30 and 26/31), and positive or mixed effectiveness was found in three-quarters of the studies (23/30). However, there were several limitations of the agents highlighted in specific qualitative feedback.ConclusionsThe studies generally reported positive or mixed evidence for the effectiveness, usability, and satisfactoriness of the conversational agents investigated, but qualitative user perceptions were more mixed. The quality of many of the studies was limited, and improved study design and reporting are necessary to more accurately evaluate the usefulness of the agents in health care and identify key areas for improvement. Further research should also analyze the cost-effectiveness, privacy, and security of the agents.International registered report identifier (irrid)RR2-10.2196/16934.

Project description:BackgroundConversational artificial intelligence (chatbots and dialogue systems) is an emerging tool for tobacco cessation that has the potential to emulate personalised human support and increase engagement. We aimed to determine the effect of conversational artificial intelligence interventions with or without standard tobacco cessation interventions on tobacco cessation outcomes among adults who smoke, compared to no intervention, placebo intervention or an active comparator.MethodsA comprehensive search of six databases was completed in June 2022. Eligible studies included randomised controlled trials published since 2005. The primary outcome was sustained tobacco abstinence, self-reported and/or biochemically validated, for at least 6 months. Secondary outcomes included point-prevalence abstinence and sustained abstinence of less than 6 months. Two authors independently extracted data on cessation outcomes and completed the risk of bias assessment. Random effects meta-analysis was conducted.ResultsFrom 819 studies, five randomised controlled trials met inclusion criteria (combined sample size n = 58,796). All studies differed in setting, methodology, intervention, participants and end-points. Interventions included chatbots embedded in multi- and single-component smartphone apps (n = 3), a social media-based (n = 1) chatbot, and an internet-based avatar (n = 1). Random effects meta-analysis of three studies found participants in the conversational artificial intelligence enhanced intervention were significantly more likely to quit smoking at 6-month follow-up compared to control group participants (RR = 1.29, 95% CI (1.13, 1.46), p < 0.001). Loss to follow up was generally high. Risk of bias was high overall.ConclusionWe found limited but promising evidence on the effectiveness of conversational artificial intelligence interventions for tobacco cessation. Although all studies found benefits from conversational artificial intelligence interventions, results should be interpreted with caution due to high heterogeneity. Given the rapid evolution and potential of artificial intelligence interventions, further well-designed randomised controlled trials following standardised reporting guidelines are warranted in this emerging area.

Project description:IntroductionArtificial intelligence (AI) enables machines to perform many complicated human skills which require various levels of human intelligence. In the field of medicine, AI helps physicians in making diagnoses and treatments for patients with more efficiency, accuracy, and precision. In order to prepare medical students who are the future healthcare workforce, it is important to enhance their readiness, knowledge and perception toward AI. This study aims to assess Pelita Harapan University (PHU) medical students' readiness, knowledge, and perception toward AI.MethodsA quantitative cross-sectional study was conducted to assess respondents' readiness, knowledge and perception toward AI. An online questionnaire was distributed via Google Forms to all batch of medical students. Medical Artificial Intelligence Readiness Scale for Medical Students (MAIRS-MS) questionnaire was used to evaluate AI readiness, while an adapted questionnaires was used to evaluate knowledge and perception toward AI. Data were then analyzed using IBM Statistical Package for Social Sciences (SPSS) version 23.0.ResultsA total of 650 respondents were included in this study. Most respondents were in pre-clinical phase (88%) while the remaining were in clinical phase (12%). Overall, the total mean score for AI readiness was 73.34 of 100. Respondents had a mean score 24.52 ± 5.26 of 40, 27.78 ± 4.65 of 40, 10.57 ± 2.07 of 15, and 10.47 ± 2.00 of 15 in the cognitive, ability, vision, and ethics domain respectively. Generally, respondents had sufficient knowledge and positive perception toward AI. There were also significant correlation between readiness and knowledge with gender, having studied coding previously in high school, and having family or close friends working in AI field. Social media also had a good influence on enchancing readiness in the domain of ability and ethics, and perception towards AI.ConclusionMedical students of PHU mostly showed neutral to favorable response on readiness, knowledge, and perception towards AI. Incorporating AI into high school and medical curriculum is an important step to prepare medical students' encounter and partnership with AI as the future workforce in medicine.

Dataset Information

Perception, performance, and detectability of conversational artificial intelligence across 32 university courses.

Publications

Perception, performance, and detectability of conversational artificial intelligence across 32 university courses.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets