Promises and Pitfalls of Anchoring Vignettes in Health Survey Research.
ABSTRACT: Data harmonization is a topic of growing importance to demographers, who increasingly conduct domestic or international comparative research. Many self-reported survey items cannot be directly compared across demographic groups or countries because these groups differ in how they use subjective response categories. Anchoring vignettes, already appearing in numerous surveys worldwide, promise to overcome this problem. However, many anchoring vignettes have not been formally evaluated for adherence to the key measurement assumptions of vignette equivalence and response consistency. This article tests these assumptions in some of the most widely fielded anchoring vignettes in the world: the health vignettes in the World Health Organization (WHO) Study on Global AGEing and Adult Health (SAGE) and World Health Survey (WHS) (representing 10 countries; n = 52,388), as well as similar vignettes in the Health and Retirement Study (HRS) (n = 4,528). Findings are encouraging regarding adherence to response consistency, but reveal substantial violations of vignette equivalence both cross-nationally and across socioeconomic groups. That is, members of different sociocultural groups appear to interpret vignettes as depicting fundamentally different levels of health. The evaluated anchoring vignettes do not fulfill their promise of providing interpersonally comparable measures of health. Recommendations for improving future implementations of vignettes are discussed.
Project description:Objective:Anchoring vignettes appear with growing frequency in surveys of health and aging, but little research investigates how to optimize their wording. This study experimentally tests whether mentioning specific health conditions and/or medical procedures enhances or undermines vignette validity. Methods:Three series of general health anchoring vignettes were fielded to 2,550 respondents in the Wisconsin Longitudinal Study: one mentioning no specific health conditions or procedures, one mentioning heart disease-related ones, and one mentioning diabetes-related ones. Variations on hierarchical ordered probit models were used to test whether vignette wording affected adherence to the key measurement assumptions of vignette equivalence (VE) and response consistency (RC). Results:While all vignette series showed substantial violations of VE, violations were larger (especially by sex and education) when using disease-specific texts. RC violations appeared relatively minor, but somewhat larger in disease-specific texts. Discussion:These findings suggest that more general, universal vignette texts may be preferable to ones describing highly specific conditions/procedures. The common advice to prioritize specificity and concreteness in survey texts may be misguided if sociodemographic groups differ in their familiarity or associations with the presented details. Anchoring vignettes are a potentially useful survey tool, but further efforts are needed to optimize their wording.
Project description:The research presented in this article combines mathematical derivations and empirical results to investigate effects of the nonparametric anchoring vignette approach proposed by King, Murray, Salomon, and Tandon on the reliability and validity of rating data. The anchoring vignette approach aims to correct rating data for response styles to improve comparability across individuals and groups. Vignettes are used to adjust self-assessment responses on the respondent level but entail significant assumptions: They are supposed to be invariant across respondents, and the responses to vignette prompts are supposed to be without error and strictly ordered. This article shows that these assumptions are not always met and that the anchoring vignette approach leads to higher Cronbach's alpha values and increased correlations among adjusted variables regardless of whether the assumptions of the approach are met or violated. Results suggest that the underlying assumptions and effects of the anchoring vignette approach should be carefully examined as the increased correlations and reliability estimates can be observed even for response variables that are independent random draws and uncorrelated with any other variable.
Project description:Recent studies report systematic differences in how individuals categorize the severity of identical health and work limitation vignettes. We investigate how health professionals and disability recipients characterize the severity of work limitations and whether their reporting patterns are robust to demographic, education, and health characteristics. We use the results to illustrate the potential impact of reporting heterogeneity on the distribution of work disability estimated from self-reported categorical health and disability data.Nationally representative data on anchoring disability vignettes from the 2004 Health and Retirement Study (HRS) are used to investigate how respondents with an occupation background in health and Social Security disability beneficiaries categorize work limitation vignettes. Using pain, cardiovascular health, and depression vignettes, we estimate generalized ordered probit models (N = 2,660 individuals or 39,681 person-vignette observations) that allow the severity thresholds to vary by respondent characteristics.We find that health professionals (excluding nurses) and disability recipients tend to classify identical work limitations as more severe compared to non-health professional non-disabled respondents. For disability recipients, the differences are most pronounced and particularly visible in the tails of the work limitations distribution. For health professionals, we observe smaller differences, affecting primarily the classification of mildly and moderately severe work limitations. The patterns for health professionals (excluding nurses) are robust to demographics, education, and health conditions. The greater likelihood of viewing the vignette person as more severely work limited observed among disability recipients is mostly explained by the fact that these respondents also tend to be in poorer health which itself predicts a more inclusive scale.Knowledge of reporting scales from health professionals and disabled individuals can benefit researchers in a broad range of applications in health and disability research. They may be useful as reference scales to evaluate disability survey data. Such knowledge may be beneficial when studying disability programs. Given the increasing availability of anchoring vignette data in surveys, this is a promising area for future evaluation research.
Project description:Data from self-report tools cannot be readily compared between cultures due to culturally specific ways of using a response scale. As such, anchoring vignettes have been proposed as a suitable methodology for correcting against this difference. We developed anchoring vignettes for the Big Five Inventory-44 (BFI-44) to supplement its Likert-type response options. Based on two samples (Rwanda: n = 423; Philippines: n = 143), we evaluated the psychometric properties of the measure both before and after applying the anchoring vignette adjustment. Results show that adjusted scores had better measurement properties, including improved reliability and a more orthogonal correlational structure, relative to scores based on the original Likert scale. Correlations of the Big Five Personality Factors with life satisfaction were essentially unchanged after the vignette-adjustment while correlations with counterproductive were noticeably lower. Overall, these changed findings suggest that the use of anchoring vignette methodology improves the cross-cultural comparability of self-reported personality, a finding of potential interest to the field of global workforce research and development as well as educational policymakers.
Project description:AIMS:To develop clinical descriptions (vignettes) of life with Alzheimer's disease (AD), we conducted focus groups of persons with AD (n = 14), family caregivers of persons with AD (n = 20), and clinicians who see persons with AD in their practices (n = 5). METHODS:Group participants read existing descriptions of AD and commented on the realism and comprehensibility of the descriptions. We used thematic framework analysis to code the comments into themes and develop three new vignettes to describe mild, moderate, and severe AD. RESULTS:Themes included the types of symptoms to mention in the new vignettes, plus the manner in which the vignettes should be written. Since the vignette descriptions were based on focus group participants' first-hand knowledge of AD, the descriptions can be said to demonstrate content validity. CONCLUSION:Members of the general public can read the vignettes and estimate their health-related quality-of-life (HRQoL) as if they had AD based on the vignette descriptions. This is especially important for economic evaluations of new AD medications, which require HRQoL to be assessed in a manner that persons with AD often find difficult to undertake. The vignettes will allow the general public to serve as a proxy and provide HRQoL estimates in place of persons with AD.
Project description:Overprescribing of antibiotics for acute respiratory infections (ARIs) is common. Our objective was to develop and validate a vignette-based method to estimate clinician ARI antibiotic prescribing. We surveyed physicians (n = 78) and retail clinic clinicians (n = 109) between January and September 2013. We surveyed clinicians using a set of ARI vignettes and linked the responses to electronic health record data for all ARI visits managed by these clinicians during 2012. We then created a new measure of antibiotic prescribing, the comprehensive ARI management rate. This was defined as not prescribing antibiotics for antibiotic-inappropriate diagnoses and prescribing guideline-concordant antibiotics for antibiotic-appropriate diagnoses (and also included appropriate use of streptococcal testing for the pharyngitis vignettes). We compared the vignette-based and chart-based comprehensive ARI management at the clinician level. We then identified the combination of vignettes that best predicted comprehensive ARI management rates, using a partitioning algorithm. Responses to 3 vignettes partitioned clinicians into 4 groups with chart-based comprehensive ARI management rates of 61% (n = 121), 50% (n = 47), 31% (n = 12), and 22% (n = 7). Responses to 3 clinical vignettes can identify clinicians with relatively poor quality ARI antibiotic prescribing. Vignettes may be a mechanism to target clinicians for quality improvement efforts.
Project description:This article addresses a potentially serious problem with the widely used self-rated health (SRH) survey item: that different groups have systematically different ways of using the item's response categories. Analyses based on unadjusted SRH may thus yield misleading results. The authors evaluate anchoring vignettes as a possible solution to this problem. Using vignettes specifically designed to calibrate the SRH item and data from the Wisconsin Longitudinal Study (WLS; N = 2,625), the authors show how demographic and health-related factors, including sex and education, predict differences in rating styles. Such differences, when not adjusted for statistically, may be sufficiently large to lead to mistakes in rank orderings of groups. In the present sample, unadjusted models show that women have better SRH than men, but this difference disappears in models adjusting for women's greater health-optimism. Anchoring vignettes appear a promising tool for improving intergroup comparability of SRH.
Project description:Although mental health literacy has been widely studied in adults, there are still relatively few studies on adolescent populations. In Sri Lanka, adolescents account for about one fifth of the population. Current evidence shows that most mental health problems diagnosed in adulthood begin in adolescence. There is also growing evidence that the trajectories of these disorders can be altered through early recognition and intervention. Although, help-seeking for mental health problems is known to be poor in adolescents, mental health literacy improves help-seeking. It is also known that adolescents may act as agents of change regarding mental health in their wider communities. Thus, mental health literacy in adolescents is an important aspect of community mental health initiatives. The objective of this study was to describe aspects of mental health literacy in terms of ability to recognise problems, helpful interventions, helpful referral options and outcomes in a target adolescent population in Sri Lanka. The association between socio economic variables and recognition of mental health problems was also examined.This descriptive cross sectional study used a pretested questionnaire on 1002 adolescents aged between 13 and 16, where mental health literacy was assessed using 4 case vignettes. The vignettes represented depression with suicidal ideation, social phobia, psychosis and diabetes, where the last was for comparison.The response rates for recognition as a mental health problem was 82.2% (n = 824) for the vignette depicting depression, 68.7% (n = 689) for the psychosis vignette and 62.3% (n = 623) for the social phobia vignette. "Talking to the person", was responded to as helpful by 49.9% (n = 500), for the depression vignette followed by 49.8% (n = 499) for social phobia, 39.5% (n = 396) for psychosis and 19.5% (n = 195) for the diabetes vignette. The response rate for exercise being a helpful intervention was 25% (n = 251) for the diabetes vignette, followed by 21% (n = 210) for social phobia, 18.7% (n = 187) for psychosis vignette and 18.4% (n = 184) for the depression vignette. While 70.2% (n = 704) responded that there would be benefit in seeing a doctor for the diabetes vignette, the response rates for psychosis was 48.5% (n = 486), and for both depression and social phobia it was 48.2% (n = 483). The responses for the persons in the vignettes becoming better with treatment was 81.4% (n = 816) for the diabetes, 79.5% (n = 797) for depression, 75.6% (n = 758) for psychosis and 63.4% (n = 636) for the social phobia vignette. A statistically significant association was found between the income level of the family and appropriate recognition as mental health problems, for all the 3 mental health related vignettes.The ability to recognise mental health problems, helpful interventions and outcomes in this population was comparable to those of adolescent populations in other countries, with some exceptions. The main differences were in relation to the identification and interventions in response to the psychosis and social phobia vignettes.
Project description:As virtual experiences are increasingly used in health care training and research, it is important that adequate processes are applied for developing valid scenarios. We describe the development and validation of virtual human (VH) vignettes, computer-generated scenarios with animated patients and clinical information, for a mixed-methods study regarding nurses' assessment and intervention choices for critically ill children's pain.We followed the case development and review process for high-fidelity simulation case scenarios, including the use of validated written vignettes and content experts. Forty nurses described their pain assessment and intervention choices for the newly derived VH vignettes and completed a pain questionnaire. Nurses' reports of VH vignette consistency with their professional experience and recognition of VH facial expressions were evaluated to establish face validity. Their pain ratings for the VH and written (questionnaire) vignettes were evaluated for convergent validity. Qualitative content analysis, descriptive statistics, correlations, and paired t tests were used.Most nurses (68.4%) supported vignette consistency with their professional experience. Facial expression recognition was 98.4%. Smiling children's pain was rated significantly lower than grimacing children in both VH and written vignettes. Pain was rated significantly lower for grimacing children in the VH vignettes than the written vignettes. Virtual human vignette pain ratings were strongly correlated with their written counterparts.This process was effective for developing VH vignettes that demonstrated good face validity with participants and convergent validity with written vignettes. Virtual human vignettes may be useful in studying the influence of facial actions on nurses' choices for children's pain assessment and treatment.
Project description:In our aging society, palliative care should be a standard component of health care. However, currently it is only provided to a small proportion of patients, mostly to those with cancer, and restricted to the terminal phase. Many general practitioners (GPs) say that one of their most significant challenges is to assess the right moment to start anticipatory palliative care. The "Surprise Question" (SQ1: "Would I be surprised if this patient were to die in the next 12 months"?), if answered with "no", is an easy tool to apply in identifying patients in need of palliative care. However, this tool has a low specificity. Therefore, the aim of our pilot study was to determine if adding a second, more specific "Surprise Question" (SQ2: "Would I be surprised if this patient is still alive after 12 months"?) in case SQ1 is answered in the negative, prompts GPs to plan for anticipatory palliative care.By randomization, 28 GPs in the south-eastern part of the Netherlands were allocated to three different groups. They all received a questionnaire with four vignettes, respectively representing patients with advanced organ failure (A), end stage cancer (B), frailty (C), and recently diagnosed cancer (D). GPs in the first group did not receive additional information, the second group received SQ1 after each vignette, and the third group received SQ1 and SQ2 after each vignette. We rated their answers based on essential components of palliative care (here called RADIANT score).GPs in group 3 gave higher RADIANT scores to those vignettes in which they would be surprised if the patients were still alive after 12 months. In all groups, vignette B had the highest mean RADIANT score, followed by vignettes A and C, and the lowest on vignette D. Seventy-one percent of GPs in groups 2 and 3 considered SQ1 a helpful tool, and 75% considered SQ2 helpful.This innovative pilot study indicates that the majority of GPs think SQ2 is a helpful additional tool. The combination of the two "Surprise Questions" encourages GPs to make more specific plans for anticipatory palliative care.