Dataset Information

Measuring how motivation affects information quality assessment: A gamification approach.

ABSTRACT:

Purpose

Existing research on the measurability of information quality (IQ) has delivered poor results and demonstrated low inter-rater agreement measured by Intra-Class Correlation (ICC) in evaluating IQ dimensions. Low ICC could result in a questionable interpretation of IQ. The purpose of this paper is to analyse whether assessors' motivation can facilitate ICC.

Methodology

To acquire the participants' views of IQ, we designed a survey as a gamified process. Additionally, we selected Web study to reach a broader audience. We increased the validity of the research by including a diverse set of participants (i.e. individuals with different education, demographic and social backgrounds).

Findings

The study results indicate that motivation improved the ICC of IQ on average by 0.27, demonstrating an increase in measurability from poor (0.29) to moderate (0.56). The results reveal a positive correlation between motivation level and ICC, with a significant overall increase in ICC relative to previous studies. The research also identified trends in ICC for different dimensions of IQ with the best results achieved for completeness and accuracy.

Practical implications

The work has important practical implications for future IQ research and suggests valuable guidelines. The results of this study imply that considering raters' motivation improves the measurability of IQ substantially.

Originality

Previous studies addressed ICC in IQ dimension evaluation. However, assessors' motivation has been neglected. This study investigates the impact of assessors' motivation on the measurability of IQ. Compared to the results in related work, the level of agreement achieved with the most motivated group of participants was superior.

SUBMITTER: Pozenel M

PROVIDER: S-EPMC9612527 | biostudies-literature | 2022

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Measuring how motivation affects information quality assessment: A gamification approach.

Poženel Marko M Zrnec Aljaž A Lavbič Dejan D

PloS one 20221027 10

<h4>Purpose</h4>Existing research on the measurability of information quality (IQ) has delivered poor results and demonstrated low inter-rater agreement measured by Intra-Class Correlation (ICC) in evaluating IQ dimensions. Low ICC could result in a questionable interpretation of IQ. The purpose of this paper is to analyse whether assessors' motivation can facilitate ICC.<h4>Methodology</h4>To acquire the participants' views of IQ, we designed a survey as a gamified process. Additionally, we sel ...[more]

PMID: 36301861

Similar Datasets

Project description:BackgroundProgrammatic assessment is increasingly being implemented within competency-based health professions education. In this approach a multitude of low-stakes assessment activities are aggregated into a holistic high-stakes decision on the student's performance. High-stakes decisions need to be of high quality. Part of this quality is whether an examiner perceives saturation of information when making a holistic decision. The purpose of this study was to explore the influence of narrative information in perceiving saturation of information during the interpretative process of high-stakes decision-making.MethodsIn this mixed-method intervention study the quality of the recorded narrative information was manipulated within multiple portfolios (i.e., feedback and reflection) to investigate its influence on 1) the perception of saturation of information and 2) the examiner's interpretative approach in making a high-stakes decision. Data were collected through surveys, screen recordings of the portfolio assessments, and semi-structured interviews. Descriptive statistics and template analysis were applied to analyze the data.ResultsThe examiners perceived less frequently saturation of information in the portfolios with low quality of narrative feedback. Additionally, they mentioned consistency of information as a factor that influenced their perception of saturation of information. Even though in general they had their idiosyncratic approach to assessing a portfolio, variations were present caused by certain triggers, such as noticeable deviations in the student's performance and quality of narrative feedback.ConclusionThe perception of saturation of information seemed to be influenced by the quality of the narrative feedback and, to a lesser extent, by the quality of reflection. These results emphasize the importance of high-quality narrative feedback in making robust decisions within portfolios that are expected to be more difficult to assess. Furthermore, within these "difficult" portfolios, examiners adapted their interpretative process reacting on the intervention and other triggers by means of an iterative and responsive approach.

Project description:We present primary results from the Sequencing Quality Control (SEQC) project, coordinated by the United States Food and Drug Administration. Examining Illumina HiSeq, Life Technologies SOLiD and Roche 454 platforms at multiple laboratory sites using reference RNA samples with built-in controls, we assess RNA sequencing (RNA-seq) performance for sequence discovery and differential expression profiling and compare it to microarray and quantitative PCR (qPCR) data using complementary metrics. At all sequencing depths, we discover unannotated exon-exon junctions, with >80% validated by qPCR. We find that measurements of relative expression are accurate and reproducible across sites and platforms if specific filters are used. In contrast, RNA-seq and microarrays do not provide accurate absolute measurements, and gene-specific biases are observed, for these and qPCR. Measurement performance depends on the platform and data analysis pipeline, and variation is large for transcriptlevel profiling. The complete SEQC data sets, comprising >100 billion reads (10Tb), provide unique resources for evaluating RNA-seq analyses for clinical and regulatory settings. The well-characterized reference RNA samples A (pooled cell lines) and B (human brain) from the MAQC consortium, adding spike-ins of synthetic RNA from the External RNA Control Consortium (ERCC). Samples C and D were then constructed by combining A and B in known mixing ratios, 3:1 and 1:3, respectively. All samples were distributed to several independent sites for RNA-Seq library construction and profiling by Illumina HiSeq 2000 and LifeTech SOLiD 5500 platforms. Also, vendors created their own cDNA libraries that were then distributed to each test site, in order to examine the degree of a M-bM-^@M-^\site effectM-bM-^@M-^] that was independent of the library preparation process. To support an assessment of gene models, samples A and B were also sequenced at independent sites by the Roche 454 platform, providing longer reads. For comparison to other technologies, these data were also compared to the MAQC-I Affymetrix U133 Plus2 microarray, several current microarray platforms, and also assessed by 20,801 PrimePCR reactions.

Project description:BackgroundOver 20,000 parents in the United States face the challenge of participating in decisions about whether to use life support for their infants born on the cusp of viability every year. Clinicians must help families grasp complex medical information about their baby's immediate prognosis as well as the risk for significant long-term morbidity. Patients faced with this decision want supplemental information and frequently seek medical information on the Internet. Empirical evidence about the quality of websites is lacking.ObjectiveWe sought to evaluate the quality of online information available about periviable birth and treatment options for infants born at the cusp of viability.MethodsWe read a counseling script to 20 pregnant participants that included information typically provided by perinatal and neonatal providers when periviable birth is imminent. The women were then asked to list terms they would use to search the Internet if they wanted additional information. Using these search terms, two reviewers evaluated the content of websites obtained via a Google search. We used two metrics to assess the quality of websites. The first was the DISCERN instrument, a validated questionnaire designed to assess the quality of patient-targeted health information for treatment choices. The second metric was the Essential Content Tool (ECT), a tool designed to address key components of counseling around periviable birth as outlined by professional organizations. DISCERN scores were classified as low quality if scores were 2, fair quality if scores were 3, and high quality if scores were 4 or higher. Scores of 6 or higher on the ECT were considered high quality. Interreviewer agreement was assessed by calculated kappa statistic.ResultsA total of 97 websites were reviewed. Over half (57/97, 59%) were for-profit sites, news stories, or personal blogs; 28% (27/97) were government or medical sites; and 13% (13/97) were nonprofit or advocacy sites. The majority of sites scored poorly in DISCERN questions designed to assess the reliability of information presented as well as data regarding treatment choices. Only 7% (7/97) of the websites were high quality as defined by the DISCERN tool. The majority of sites did not address the essential content defined by the ECT. Importantly, only 18% of websites (17/97) indicated that there are often a number of reasonable approaches to newborn care when faced with periviable birth. Agreement was strong, with kappa ranging from .72 to .91.ConclusionsMost information about periviable birth found on the Internet using common search strategies is of low quality. News stories highlighting positive outcomes are disproportionately represented. Few websites discuss comfort care or how treatment decisions impact quality of life.

Dataset Information

Measuring how motivation affects information quality assessment: A gamification approach.

Purpose

Methodology

Findings

Practical implications

Originality

Publications

Measuring how motivation affects information quality assessment: A gamification approach.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets