Dataset Information

Evaluation of an online text simplification editor using manual and automated metrics for perceived and actual text difficulty.

ABSTRACT:

Objective

Simplifying healthcare text to improve understanding is difficult but critical to improve health literacy. Unfortunately, few tools exist that have been shown objectively to improve text and understanding. We developed an online editor that integrates simplification algorithms that suggest concrete simplifications, all of which have been shown individually to affect text difficulty.

Materials and methods

The editor was used by a health educator at a local community health center to simplify 4 texts. A controlled experiment was conducted with community center members to measure perceived and actual difficulty of the original and simplified texts. Perceived difficulty was measured using a Likert scale; actual difficulty with multiple-choice questions and with free recall of information evaluated by the educator and 2 sets of automated metrics.

Results

The results show that perceived difficulty improved with simplification. Several multiple-choice questions, measuring actual difficulty, were answered more correctly with the simplified text. Free recall of information showed no improvement based on the educator evaluation but was better for simplified texts when measured with automated metrics. Two follow-up analyses showed that self-reported education level and the amount of English spoken at home positively correlated with question accuracy for original texts and the effect disappears with simplified text.

Discussion

Simplifying text is difficult and the results are subtle. However, using a variety of different metrics helps quantify the effects of changes.

Conclusion

Text simplification can be supported by algorithmic tools. Without requiring tool training or linguistic knowledge, our simplification editor helped simplify healthcare related texts.

SUBMITTER: Leroy G

PROVIDER: S-EPMC9155254 | biostudies-literature | 2022 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evaluation of an online text simplification editor using manual and automated metrics for perceived and actual text difficulty.

Leroy Gondy G Kauchak David D Haeger Diane D Spegman Douglas D

JAMIA open 20220530 2

<h4>Objective</h4>Simplifying healthcare text to improve understanding is difficult but critical to improve health literacy. Unfortunately, few tools exist that have been shown objectively to improve text and understanding. We developed an online editor that integrates simplification algorithms that suggest concrete simplifications, all of which have been shown individually to affect text difficulty.<h4>Materials and methods</h4>The editor was used by a health educator at a local community healt ...[more]

PMID: 35663117

Similar Datasets

Project description:PurposeLow patient health literacy has been associated with cost increases in medicine because it contributes to inadequate care. Providing explanatory text is a convenient approach to distribute medical information and increase health literacy. Unfortunately, writing text that is easily understood is challenging. This work tests two text features for their impact on understanding: lexical simplification and coherence enhancement.MethodsA user study was conducted to test the features' effect on perceived and actual text difficulty. Individual sentences were used to test perceived difficulty. Using a 5-point Likert scale, participants compared eight pairs of original and simplified sentences. Abstracts were used to test actual difficulty. For each abstract, four versions were created: original, lexically simplified, coherence enhanced, and lexically simplified and coherence enhanced. Using a mixed design, one group of participants worked with the original and lexically simplified documents (no coherence enhancement) while a second group worked with the coherence enhanced versions. Actual difficulty was measured using a Cloze measure and multiple-choice questions.ResultsUsing Amazon's Mechanical Turk, 200 people participated of which 187 qualified based on our data qualification tests. A paired-samples t-test for the sentence ratings showed a significant reduction in difficulty after lexical simplification (p<.001). Results for actual difficulty are based on the abstracts and associated tasks. A two-way ANOVA for the Cloze test showed no effect of coherence enhancement but a main effect for lexical simplification, with the simplification leading to worse scores (p=.004). A follow-up ANOVA showed this effect exists only for function words when coherence was not enhanced (p=.008). In contrast, a two-way ANOVA for answering multiple-choice questions showed a significant beneficial effect of coherence enhancement (p=.003) but no effect of lexical simplification.ConclusionsLexical simplification reduced the perceived difficulty of texts. Coherence enhancement reduced the actual difficulty of text when measured using multiple-choice questions. However, the Cloze measure results showed that lexical simplification can negatively impact the flow of the text.

Dataset Information

Evaluation of an online text simplification editor using manual and automated metrics for perceived and actual text difficulty.

Objective

Materials and methods

Results

Discussion

Conclusion

Publications

Evaluation of an online text simplification editor using manual and automated metrics for perceived and actual text difficulty.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets