The effect of administration mode on CAHPS survey response rates and results: A comparison of mail and web-based approaches.
ABSTRACT: OBJECTIVE:The objective of this study was to compare response rates, respondents' characteristics, and substantive results for CAHPS surveys administered using web and mail protocols. DATA SOURCES:Patients who had one or more primary care visits in the preceding 6 months. STUDY DESIGN/DATA COLLECTION METHODS:Patients for whom primary care practices had email addresses were randomized to one of four survey administration protocols: web via a portal invitation; web via an email invitation; combination of web and mail; and mail only. Another sample of patients without known email addresses was surveyed by mail. Samples of nonrespondents to the Internet and mail protocols were surveyed by telephone. PRINCIPAL FINDINGS:Response rates to surveys administered using the Internet protocols were lower than for the surveys administered by mail (20 percent vs over 40 percent). However, characteristics of respondents and survey answers were very similar across protocols. Respondents without email addresses were older, less educated, and more likely to be male than those with email addresses, and there were a few differences in their responses. There was little evidence of nonresponse bias in either the mail or web protocols. CONCLUSION:In this well-educated patient population, web protocols had lower response rates, but substantive results very similar to those from mail protocols.
Project description:OBJECTIVE:The standard data-collection procedure in the Norwegian national patient experience survey programme is post-discharge mail surveys, which include a pen-and-paper questionnaire with the option to answer electronically. A purely electronic protocol has not previously been explored in Norway. The aim of this study was to compare response rates, background characteristics, data quality and main study results for a survey of patient experiences with general practitioners (GPs) administered by the standard mail data-collection procedure and a web-based approach. DESIGN:Cross-sectional survey. SETTING:GP offices in Norway. PARTICIPANTS:The sample consisted of 6999 patients aged 16 years and older registered with a GP in November 2018. INTERVENTION:Based on a three-stage sampling design, 6999 patients of GPs aged 16 or older were randomised to one of two survey administration protocols: Group A, who were mailed an invitation with both a pen-and-paper including an electronic response option (n=4999) and Group B, who received an email invitation with electronic response option (n=2000). MAIN OUTCOME MEASURES:Response rates, background characteristics, data quality and main study results. RESULTS:The response rate was markedly higher for the mail survey (42.6%) than for the web-based survey (18.3%). A few of the background variables differed significantly between the two groups, but the data quality and patient-reported experiences were similar. CONCLUSIONS:Web-based surveys are faster and less expensive than standard mail surveys, but their low response rates and coverage problems threaten their usefulness and legitimacy. Initiatives to increase response rates for web-based data collection and strategies for tailoring data collection to different groups should be key elements in future research.
Project description:The goal of this study was to determine response rates and associated costs of different survey methods among colorectal cancer (CRC) survivors.We assembled a cohort of 16,212 individuals diagnosed with CRC (2010-2014) from six health plans, and randomly selected 4000 survivors to test survey response rates across four mixed-mode survey administration protocols (in English and Spanish): arm 1, mailed survey with phone follow-up; arm 2, interactive voice response (IVR) followed by mail; arm 3; email linked to web-based survey with mail follow-up; and arm 4, email linked to web-based survey followed by IVR.Our overall response rate was 50.2%. Arm 1 had the highest response rate (59.9%), followed by arm 3 (51.9%), arm 2 (51.2%), and arm 4 (37.9%). Response rates were higher among non-Hispanic whites in all arms than other racial/ethnic groups (p < 0.001), among English (51.5%) than Spanish speakers (36.4%) (p < 0.001), and among higher (53.7%) than lower (41.4%) socioeconomic status (p < 0.001). Survey arms were roughly comparable in cost, with a difference of only 8% of total costs between the most (arm 2) and least (arm 3) expensive arms.Mailed surveys followed by phone calls achieved the highest response rate; email invitations and online surveys cost less per response. Electronic methods, even among those with email availability, may miss important populations including Hispanics, non-English speakers, and those of lower socioeconomic status.Our results demonstrate effective methods for capturing patient-reported outcomes, inform the relative benefits/disadvantages of the different methods, and identify future research directions.
Project description:BACKGROUND:Low participation rates are one of the most serious disadvantages of Web-based studies. It is necessary to develop effective strategies to improve participation rates to obtain sufficient data. OBJECTIVE:The objective of this trial was to investigate the effect of emphasizing the incentive in the subject line of the invitation email and the day of the week of sending the invitation email on the participation rate in a Web-based trial. METHODS:We conducted a 2×2 factorial design randomized controlled trial. We contacted 2000 primary care physicians from members of the Japan Primary Care Association in January 2017 and randomly allocated them to 1 of 4 combinations of 2 subject lines (presence or absence of an emphasis on a lottery for an Amazon gift card worth 3000 yen or approximately US $30) and 2 delivery days (sending the invitation email on Tuesday or Friday). The primary outcome was the response rate defined as the number of participants answering the first page of the questionnaire divided by the number of invitation emails delivered. All outcomes were collected between January 17, 2017, and February 8, 2017. RESULTS:We analyzed data from 1943 out of 2000 participants after excluding those whose email addresses were invalid. The overall response rate was 6.3% (123/1943). There was no significant difference in the response rates between the 2 groups regarding incentive in the subject line: the risk ratio was 1.12 (95% CI 0.80 to 1.58) and the risk difference was 0.7% (95% CI -1.5% to 2.9%). Similarly, there was no significant difference in the response rates between the 2 groups regarding sending the email on Tuesday or Friday: the risk ratio was 0.98 (95% CI 0.70 to 1.38) and the risk difference was -0.1% (95% CI -2.3% to 2.1%). CONCLUSIONS:Neither emphasizing the incentive in the subject line of the invitation email nor varying the day of the week the invitation email was sent led to a meaningful increase in response rates in a Web-based trial with primary care physicians. TRIAL REGISTRATION:University Hospital Medical Information Network Clinical Trials Registry UMIN000025317; https://upload.umin.ac.jp/cgi-open-bin/ctr_e/ctr_view.cgi?recptno=R000029121 (Archived by WebCite at http://www.webcitation. org/6wOo1jl9t).
Project description:BACKGROUND:Paper questionnaires have traditionally been the first choice for data collection in research. However, declining response rates over the past decade have increased the risk of selection bias in cross-sectional studies. The growing use of the Internet offers new ways of collecting data, but trials using Web-based questionnaires have so far seen mixed results. A secure, online digital mailbox (e-Boks) linked to a civil registration number became mandatory for all Danish citizens in 2014 (exemption granted only in extraordinary cases). Approximately 89% of the Danish population have a digital mailbox, which is used for correspondence with public authorities. OBJECTIVE:We aimed to compare response rates, completeness of data, and financial costs for different invitation methods: traditional surface mail and digital mail. METHODS:We designed a cross-sectional comparative study. An invitation to participate in a survey on help-seeking behavior in out-of-hours care was sent to two groups of randomly selected citizens from age groups 30-39 and 50-59 years and parents to those aged 0-4 years using either traditional surface mail (paper group) or digital mail sent to a secure online mailbox (digital group). Costs per respondent were measured by adding up all costs for handling, dispatch, printing, and work salary and then dividing the total figure by the number of respondents. Data completeness was assessed by comparing the number of missing values between the two methods. Socioeconomic variables (age, gender, family income, education duration, immigrant status, and job status) were compared both between respondents and nonrespondents and within these groups to evaluate the degree of selection bias. RESULTS:A total 3600 citizens were invited in each group; 1303 (36.29%) responded to the digital invitation and 1653 (45.99%) to the paper invitation (difference 9.66%, 95% CI 7.40-11.92). The costs were €1.51 per respondent for the digital group and €15.67 for paper group respondents. Paper questionnaires generally had more missing values; this was significant in five of 17 variables (P<.05). Substantial differences were found in the socioeconomic variables between respondents and nonrespondents, whereas only minor differences were seen within the groups of respondents and nonrespondents. CONCLUSIONS:Although we found lower response rates for Web-based invitations, this solution was more cost-effective (by a factor of 10) and had slightly lower numbers of missing values than questionnaires sent with paper invitations. Analyses of socioeconomic variables showed almost no difference between nonrespondents in both groups, which could imply that the lower response rate in the digital group does not necessarily increase the level of selection bias. Invitations to questionnaire studies via digital mail may be an excellent option for collecting research data in the future. This study may serve as the foundational pillar of digital data collection in health care research in Scandinavia and other countries considering implementing similar systems.
Project description:This study examines the two-year follow up (data collected in 2016 at modal age 21/22) of an original mixed-mode longitudinal survey experiment (data collected at modal age 19/20 in 2014). The study compares participant retention in the experimental conditions to retention in the standard Monitoring the Future (MTF) control condition (participants who completed an in-school baseline survey in 12th grade in 2012 or 2013 and were selected to participate in the first follow-up survey by mail in 2014, N=2,451). A supplementary sample who completed the 12th grade baseline survey in 2012 or 2013 but were not selected to participate in the main MTF follow-up (N=4,950) were recruited and randomly assigned to one of three experimental conditions: 1: Mail Push, 2: Web Push, 3: Web Push + Email in 2014 and again in 2016. Results from the first experiment indicated that Condition 3 (Web Push + Email) was promising based on similar response rates and lower costs (Patrick et al. 2018). The current study examines how experimental condition and type of 2014 response were associated with response in 2016, the extent to which response mode and device type changed from 2014 to 2016, and cumulative cost comparisons across conditions. Results indicated that responding via web in 2014 was associated with greater odds of participation again in 2016 regardless of condition; respondents tended to respond in the same mode although the "push" condition did move respondents toward web over paper; device type varied between waves; and the cumulative cost savings of Web Push + Email grew larger compared to the MTF Control. The web push strategy is therefore promising for maintaining respondent engagement while reducing cost.
Project description:BACKGROUND:Internet has been broadly employed as a facilitator for epidemiological surveys, as a way to provide a more economical and practical alternative to traditional survey modes. A current trend in survey research is to combine Web-based surveys with other survey modes by offering the participant the possibility of choosing his/her preferred response method (i.e. mixed-mode approach). However, studies have also demonstrated that the use of different survey modes may produce different responses to the same questions, posing potential challenges on the use of mixed-mode approaches. METHODS:In this paper, we have implemented a statistical comparison between mixed-mode survey responses collected via mail (i.e. paper) and Web methods obtained from a cross-sectional study in non-urban areas of Denmark. Responses provided by mail and Web participants were compared in terms of: 1) the impact of reminder letters in increasing response rates; 2) differences in socio-demographic characteristics between response groups; 3) changes on the likelihood of reporting health symptoms and negative attitudes towards environmental stressors. Comparisons were mainly performed by two sample t-test, Pearson's Chi-squared test and multinomial logistic regression models. RESULTS:Among 3104 contacted households, 1066 residents decided to participate on the study. Out of those, 971 selected to respond via mail, whereas 275 preferred the Web method. The majority of socio-demographic characteristics between these two groups of respondents were shown to be statistically different. The use of mailed surveys increased the likelihood of reporting health symptoms and negative attitudes towards environmental stressors, even after controlling for demographic characteristics. Furthermore, the use of reminder letters had a higher positive impact in increasing responses of Web surveys when compared to mail surveys. CONCLUSIONS:Our main findings suggest that the use of mail and Web surveys may produce different responses to the same questions posed to participants, but, at the same time, may reach different groups of respondents, given that the overall characteristics of both groups considerably differ. Therefore, the tradeoff between using mixed-mode survey as a way to increase response rate and obtaining undesirable measurement changes may be attentively considered in future survey studies.
Project description:Incidence of acute respiratory infections (ARI) and gastrointestinal infections (GII) are difficult to assess due to high frequency episodes, limited severity and short duration. Retrospective assessments therefore are particularly prone to recall bias, while prospective assessment with conventional questionnaires requires high discipline from participants which is difficult to maintain over longer time periods. Web-based questionnaires (WQ) allow integration of a recall system and thus carry the potential to prospectively capture acute infections. We investigated the feasibility of a weekly WQ assessing symptoms of ARI and GII among participants of the German National Cohort (GNC).In the study centres Hamburg and Bremen of the GNC participants of the Pretest 1 phase (September to November 2011) were invited to additionally take part in this feasibility study testing the WQ. Every Monday participants received an e-mail, containing a link to the WQ, asking for occurrence of ARI or GII symptoms during the past 7 days. The study took place from the beginning of February until mid-July 2012. We calculated the overall proportion of participation, weekly participation and the number of weekly reports per participant and we estimated incidences of ARI, ILI and GII.Of 200 Pretest 1 participants 171 (86?%) reported having an email address and thus were eligible for the web-based study. A total of 167 (98?%) agreed to participate. Participants of the web-based study were younger and better educated than non-participants. Access to Internet decreased with increasing age. Of the 167 participants in the feasibility study, 144 (86?%) responded at least once during the study period of 23 weeks, 5 persons (3?%) had non-functioning email addresses and 18 (11?%) did not respond at all. The weekly response varied between 62?% and 81?%, the median was 74?% (IQR: 71-77?%). Weekly median reports per person were 20 (IQR: 14-22; range 1-23). More than 90?% of participants responded during the first 3 days. The following mean incidence rates were found: ARI, 12?%; ILI, 0.49?%; and GII, 3?%.Use of WQ in prospective studies seems well possible, as Internet access is frequent among study participants and major technical problems did not occur. We observed high participation during the study period of 6 months and low drop out numbers. Participants of the web-based study were slightly younger and better educated than non-participants, so selection bias is possible and must be kept in mind when discussing generalizability of the results.
Project description:Positive attitude toward evidence-based practice (EBP) principles in healthcare education may be one of the first steps for motivating a healthcare professional student to later apply EBP principles in clinical decision-making. The objectives for this project were to pilot an international web-based survey of chiropractic students and to describe student attitudes, behaviors, and knowledge about EBP principles.We used SurveyMonkey™ to develop our survey based on an existing questionnaire used to measure basic knowledge, skills and beliefs about EBP among allied healthcare professionals and CAM practitioners. We invited 26 chiropractic educational institutions teaching in English and accredited by official organizations to participate. Academic officials and registrars at participating institutions forwarded an invitation email and two reminders to students between July and September 2010. The invitation contained a link to the 38-item web-based questionnaire. Descriptive statistics were performed for analysis.Fourteen institutions from Australia, Canada, US, Denmark and New Zealand participated. Among an estimated 7,142 student recipients of invitation letters, 674 participated in the survey for an estimated response rate of 9.4%. Most respondents reported having access to medical/healthcare literature through the internet, but only 11% read literature every week and 21% did not read literature at all. Respondents generally agreed that the use of research evidence in chiropractic was important. Although 76% of respondents found it easy to understand research evidence and 81% had some level of confidence assessing the general worth of research articles, 71% felt they needed more training in EBP to be able to apply evidence in chiropractic care. Respondents without previous training in research methods had lower confidence in assessing published papers. While more than 60% marked the correct answer for two knowledge items, the mean number of correct answers to the five knowledge questions was 1.3 (SD 0.9).Although it is feasible to conduct an international web survey of chiropractic students, significant stakeholder participation is important to improve response rates. Students had relatively positive attitudes toward EBP. However, participants felt they needed more training in EBP and based on the knowledge questions they may need further training about basic research concepts.
Project description:Telemedicine is an increasingly recognized option for cost-effective management of chronic conditions. We surveyed Sleep Clinic patients about their experiences and preferences regarding different forms of telemedicine. Adult Sleep Clinic patients seen between 2009 and 2011 received a brief survey either by postal mail (n = 156) or, for those with an available email address, electronically (n = 282). The overall response rate was 28.1% (n = 123 responses), with email response rates being higher than postal mail responses. The most commonly reported barriers to in-person physician visits were parking cost (44%), time away from work/school (34%), and cost of gas (26%). Whereas 89% of respondents indicated using telephone and 55% of respondents indicated using email to communicate with providers, none reported experience with video telemedicine. Despite this lack of experience, over 60% reported feeling comfortable or willing to try it. Of those who were uncomfortable about video telemedicine, the two main reasons were that in-person visits feel more natural (48%) and that the doctor might need to perform an examination (24%). More than half of respondents reported willingness to pay a copay for a video visit. Video telemedicine represents a feasible option for chronic sleep disorders management.
Project description:To examine how different response scales, methods of survey administration, and survey format affect responses to the CAHPS (Consumer Assessment of Healthcare Providers and Systems) Clinician and Group (CG-CAHPS) survey.A total of 6,500 patients from a university health center were randomly assigned to receive the following: standard 12-page mail surveys using 4-category or 6-category response scales (on CG-CAHPS composite items), telephone surveys using 4-category or 6-category response scales, or four-page mail surveys.A total of 3,538 patients completed surveys. Composite score means and provider-level reliabilities did not differ between respondents receiving 4-category or 6-category response scale surveys or between 12-page and four-page mail surveys. Telephone respondents gave more positive responses than mail respondents.We recommend using 4-category response scales and the four-page mail CG-CAHPS survey.