Dataset Information

An empirical appraisal of eLife's assessment vocabulary.

ABSTRACT: Research articles published by the journal eLife are accompanied by short evaluation statements that use phrases from a prescribed vocabulary to evaluate research on 2 dimensions: importance and strength of support. Intuitively, the prescribed phrases appear to be highly synonymous (e.g., important/valuable, compelling/convincing) and the vocabulary's ordinal structure may not be obvious to readers. We conducted an online repeated-measures experiment to gauge whether the phrases were interpreted as intended. We also tested an alternative vocabulary with (in our view) a less ambiguous structure. A total of 301 participants with a doctoral or graduate degree used a 0% to 100% scale to rate the importance and strength of support of hypothetical studies described using phrases from both vocabularies. For the eLife vocabulary, most participants' implied ranking did not match the intended ranking on both the importance (n = 59, 20% matched, 95% confidence interval [15% to 24%]) and strength of support dimensions (n = 45, 15% matched [11% to 20%]). By contrast, for the alternative vocabulary, most participants' implied ranking did match the intended ranking on both the importance (n = 188, 62% matched [57% to 68%]) and strength of support dimensions (n = 201, 67% matched [62% to 72%]). eLife's vocabulary tended to produce less consistent between-person interpretations, though the alternative vocabulary still elicited some overlapping interpretations away from the middle of the scale. We speculate that explicit presentation of a vocabulary's intended ordinal structure could improve interpretation. Overall, these findings suggest that more structured and less ambiguous language can improve communication of research evaluations.

SUBMITTER: Hardwicke TE

PROVIDER: S-EPMC11340897 | biostudies-literature | 2024 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

An empirical appraisal of eLife's assessment vocabulary.

Hardwicke Tom E TE Schiavone Sarah R SR Clarke Beth B Vazire Simine S

PLoS biology 20240822 8

Research articles published by the journal eLife are accompanied by short evaluation statements that use phrases from a prescribed vocabulary to evaluate research on 2 dimensions: importance and strength of support. Intuitively, the prescribed phrases appear to be highly synonymous (e.g., important/valuable, compelling/convincing) and the vocabulary's ordinal structure may not be obvious to readers. We conducted an online repeated-measures experiment to gauge whether the phrases were interpreted ...[more]

PMID: 39172747

Similar Datasets

Project description:PurposeThis study explored approaches for measuring vocabulary among bilingual children with varying levels of proficiency in Spanish and English.MethodOne hundred fifteen kindergarten and first-grade Spanish-English-speaking children completed measures of vocabulary and sentence repetition in Spanish and English. Scores were derived from their responses to the vocabulary measure: Spanish-only vocabulary, English-only vocabulary, conceptual vocabulary, and total vocabulary. Best language sentence repetition was also obtained. Using both visualization of data and statistical analysis, we tested for potential associations between children's relative language skills in Spanish and English and the scores they received on each of the vocabulary metrics.ResultsParticipants' single-language vocabulary scores were linearly associated with their relative language scores. Higher relative Spanish language skills corresponded with higher Spanish-only vocabulary scores, and higher English language skills corresponded with higher English-only vocabulary scores. A quadratic association between children's relative language and their conceptual vocabulary scores was observed. Children with more balanced skills in Spanish and English received lower scores for conceptual vocabulary. No association between total vocabulary and relative language was observed.ConclusionsResults revealed evidence of differential test bias for single-language vocabulary scores and conceptual vocabulary scores. Spanish-only vocabulary underestimated knowledge of participants with higher English proficiency, whereas English-only vocabulary underestimated knowledge of participants with higher Spanish proficiency. Conceptual scoring yielded lower values for participants with relatively balanced proficiency in Spanish and English. There is need for further consideration of score and test functioning across the full continuum of bilinguals with dynamic proficiencies in each of their languages.Supplemental materialhttps://doi.org/10.23641/asha.23796330.

Project description:IntroductionThe rapid spread of drug resistance is forcing standard treatment guidelines (STGs) to become more appropriate with due consideration of the evidence on the antimicrobial resistance (AMR) situation in Ethiopia. Therefore, we aimed to assess the local AMR recommendations in the STGs for empirical antibacterial prescriptions for the five common infectious syndromes. We also determined the quality of AMR reviews conducted in the country.MethodsWe conducted a review of the STGs used in the health centers, general hospitals, and primary hospitals in Ethiopia and assessed the AMR recommendations in STGs for empirical antibacterial prescriptions for community-acquired pneumonia (CAP), urinary tract infection (UTI), tonsillopharyngitis, acute otitis media (AOM), and bacterial dysentery. Next, we performed an overview of AMR reviews published in Ethiopia. We used the MEDLINE/PubMed, Embase, Cochrane Library, and Google Scholar databases to identify AMR reviews. The review characteristics were extracted. We also evaluated the quality of each included AMR review using a measurement tool to assess the systematic review scale (AMSTAR 2).ResultsA total of 6 STGs and 12 AMR reviews conducted in the country were included. The choice of empirical antibacterials for similar infectious syndromes (and editions) was comparable across the three levels of the health care system. None of the STGs evaluated included the local AMR recommendations for empirical antibacterial prescriptions for five common infectious syndromes. Of all the AMR reviews included, 75% had low and below methodologic quality, and none had a high-quality score using the AMSTAR 2 tool.ConclusionStandard treatment guidelines did not consider local AMR recommendations for empirically prescribing antibacterials for common infectious syndromes. The AMR reviews published in the country produced poor methodologic quality evidence for clinical applications. This highlights the need to improve the methodologic quality to provide the best available evidence for clinical decision-making and curb the ongoing AMR in Ethiopia.Trial registrationRetrospectively registered (15/07/2020).

Project description:BackgroundEpidemiological studies may be subject to selective reporting, but empirical evidence thereof is limited. We empirically evaluated the extent of selection of significant results and large effect sizes in a large sample of recent articles.Methods and findingsWe evaluated 389 articles of epidemiological studies that reported, in their respective abstracts, at least one relative risk for a continuous risk factor in contrasts based on median, tertile, quartile, or quintile categorizations. We examined the proportion and correlates of reporting statistically significant and nonsignificant results in the abstract and whether the magnitude of the relative risks presented (coined to be consistently > or =1.00) differs depending on the type of contrast used for the risk factor. In 342 articles (87.9%), > or =1 statistically significant relative risk was reported in the abstract, while only 169 articles (43.4%) reported > or =1 statistically nonsignificant relative risk in the abstract. Reporting of statistically significant results was more common with structured abstracts, and was less common in US-based studies and in cancer outcomes. Among 50 randomly selected articles in which the full text was examined, a median of nine (interquartile range 5-16) statistically significant and six (interquartile range 3-16) statistically nonsignificant relative risks were presented (p = 0.25). Paradoxically, the smallest presented relative risks were based on the contrasts of extreme quintiles; on average, the relative risk magnitude was 1.41-, 1.42-, and 1.36-fold larger in contrasts of extreme quartiles, extreme tertiles, and above-versus-below median values, respectively (p < 0.001).ConclusionsPublished epidemiological investigations almost universally highlight significant associations between risk factors and outcomes. For continuous risk factors, investigators selectively present contrasts between more extreme groups, when relative risks are inherently lower.

Dataset Information

An empirical appraisal of eLife's assessment vocabulary.

Publications

An empirical appraisal of eLife's assessment vocabulary.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets