Dataset Information

Using ChatGPT for human-computer interaction research: a primer.

ABSTRACT: ChatGPT could serve as a tool for text analysis within the field of Human-Computer Interaction, though its validity requires investigation. This study applied ChatGPT to: (1) textbox questionnaire responses on nine augmented-reality interfaces, (2) interview data from participants who experienced these interfaces in a virtual simulator, and (3) transcribed think-aloud data of participants who viewed a real painting and its replica. Using a hierarchical approach, ChatGPT produced scores or summaries of text batches, which were then aggregated. Results showed that (1) ChatGPT generated sentiment scores of the interfaces that correlated extremely strongly (r > 0.99) with human rating scale outcomes and with a rule-based sentiment analysis method (criterion validity). Additionally, (2) by inputting automatically transcribed interviews to ChatGPT, it provided meaningful meta-summaries of the qualities of the interfaces (face validity). One meta-summary analysed in depth was found to have substantial but imperfect overlap with a content analysis conducted by an independent researcher (criterion validity). Finally, (3) ChatGPT's summary of the think-aloud data highlighted subtle differences between the real painting and the replica (face validity), a distinction corresponding with a keyword analysis (criterion validity). In conclusion, our research indicates that, with appropriate precautions, ChatGPT can be used as a valid tool for analysing text data.

SUBMITTER: Tabone W

PROVIDER: S-EPMC10498031 | biostudies-literature | 2023 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Using ChatGPT for human-computer interaction research: a primer.

Tabone Wilbert W de Winter Joost J

Royal Society open science 20230913 9

ChatGPT could serve as a tool for text analysis within the field of Human-Computer Interaction, though its validity requires investigation. This study applied ChatGPT to: (1) textbox questionnaire responses on nine augmented-reality interfaces, (2) interview data from participants who experienced these interfaces in a virtual simulator, and (3) transcribed think-aloud data of participants who viewed a real painting and its replica. Using a hierarchical approach, ChatGPT produced scores or summar ...[more]

PMID: 37711151

Similar Datasets

Project description:BackgroundArtificial intelligence (AI) has many applications in various aspects of our daily life, including health, criminal, education, civil, business, and liability law. One aspect of AI that has gained significant attention is natural language processing (NLP), which refers to the ability of computers to understand and generate human language.ObjectiveThis study aims to examine the potential for, and concerns of, using AI in scientific research. For this purpose, high-impact research articles were generated by analyzing the quality of reports generated by ChatGPT and assessing the application's impact on the research framework, data analysis, and the literature review. The study also explored concerns around ownership and the integrity of research when using AI-generated text.MethodsA total of 4 articles were generated using ChatGPT, and thereafter evaluated by 23 reviewers. The researchers developed an evaluation form to assess the quality of the articles generated. Additionally, 50 abstracts were generated using ChatGPT and their quality was evaluated. The data were subjected to ANOVA and thematic analysis to analyze the qualitative data provided by the reviewers.ResultsWhen using detailed prompts and providing the context of the study, ChatGPT would generate high-quality research that could be published in high-impact journals. However, ChatGPT had a minor impact on developing the research framework and data analysis. The primary area needing improvement was the development of the literature review. Moreover, reviewers expressed concerns around ownership and the integrity of the research when using AI-generated text. Nonetheless, ChatGPT has a strong potential to increase human productivity in research and can be used in academic writing.ConclusionsAI-generated text has the potential to improve the quality of high-impact research articles. The findings of this study suggest that decision makers and researchers should focus more on the methodology part of the research, which includes research design, developing research tools, and analyzing data in depth, to draw strong theoretical and practical implications, thereby establishing a revolution in scientific research in the era of AI. The practical implications of this study can be used in different fields such as medical education to deliver materials to develop the basic competencies for both medicine students and faculty members.

Project description:In order to improve the efficiency of teaching and learning in Colleges and Universities (CAUs), this work combines the Browser/Server (B/S) framework with Model View Presenter (MVP) technology to build a college student-oriented micro-video teaching system based on Deep Learning (DL) and Human-Computer Interaction (HCI) technology. Firstly, it makes an in-depth analysis of the problems in the classroom teaching of Chinese CAUs. Three functional modules are designed for the micro-video online teaching platform: video management, user learning, and system management. Then, it uses MVP technology to analyze the use-cases of these three functional modules in detail. Based on this, the micro-video online teaching platform is designed using the B/S framework. The teaching platform interface layer realizes the HCI between the platform and users. The business logic layer responds to the user requests submitted and returns the processing results to the interface layer. Finally, the function test and stress test of each module of the micro-video online teaching platform is carried out. The test results show that the response time of the proposed micro-video teaching platform increases with the number of users. Under the peak concurrent users, the system response time is 6 s, without abnormalities. Meanwhile, the proposed teaching platform has improved students' satisfaction with classroom teaching by nearly 15% and improved the compactness of the college classroom by nearly 12%. When the number of virtual users increases and the number of services increases linearly, the Random Access Memory and Central Processing Unit growth rate is significantly lower than that of the number of services. These outcomes indicate that many system resources are reused, and the system has good scalability, which can meet users' needs for the network video teaching system. The proposed teaching platform provides a new idea for applying DL and HCI technology in researching college students' micro-video teaching mode.

Project description:BackgroundData analysis approaches such as qualitative content analysis are notoriously time and labor intensive because of the time to detect, assess, and code a large amount of data. Tools such as ChatGPT may have tremendous potential in automating at least some of the analysis.ObjectiveThe aim of this study was to explore the utility of ChatGPT in conducting qualitative content analysis through the analysis of forum posts from people sharing their experiences on reducing their sugar consumption.MethodsInductive and deductive content analysis were performed on 537 forum posts to detect mechanisms of behavior change. Thorough prompt engineering provided appropriate instructions for ChatGPT to execute data analysis tasks. Data identification involved extracting change mechanisms from a subset of forum posts. The precision of the extracted data was assessed through comparison with human coding. On the basis of the identified change mechanisms, coding schemes were developed with ChatGPT using data-driven (inductive) and theory-driven (deductive) content analysis approaches. The deductive approach was informed by the Theoretical Domains Framework using both an unconstrained coding scheme and a structured coding matrix. In total, 10 coding schemes were created from a subset of data and then applied to the full data set in 10 new conversations, resulting in 100 conversations each for inductive and unconstrained deductive analysis. A total of 10 further conversations coded the full data set into the structured coding matrix. Intercoder agreement was evaluated across and within coding schemes. ChatGPT output was also evaluated by the researchers to assess whether it reflected prompt instructions.ResultsThe precision of detecting change mechanisms in the data subset ranged from 66% to 88%. Overall κ scores for intercoder agreement ranged from 0.72 to 0.82 across inductive coding schemes and from 0.58 to 0.73 across unconstrained coding schemes and structured coding matrix. Coding into the best-performing coding scheme resulted in category-specific κ scores ranging from 0.67 to 0.95 for the inductive approach and from 0.13 to 0.87 for the deductive approaches. ChatGPT largely followed prompt instructions in producing a description of each coding scheme, although the wording for the inductively developed coding schemes was lengthier than specified.ConclusionsChatGPT appears fairly reliable in assisting with qualitative analysis. ChatGPT performed better in developing an inductive coding scheme that emerged from the data than adapting an existing framework into an unconstrained coding scheme or coding directly into a structured matrix. The potential for ChatGPT to act as a second coder also appears promising, with almost perfect agreement in at least 1 coding scheme. The findings suggest that ChatGPT could prove useful as a tool to assist in each phase of qualitative content analysis, but multiple iterations are required to determine the reliability of each stage of analysis.

Dataset Information

Using ChatGPT for human-computer interaction research: a primer.

Publications

Using ChatGPT for human-computer interaction research: a primer.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets