Patient and public engagement in research and health system decision making: A systematic review of evaluation tools.
ABSTRACT: BACKGROUND:Patient and public engagement is growing, but evaluative efforts remain limited. Reviews looking at evaluation tools for patient engagement in individual decision making do exist, but no similar articles in research and health systems have been published. OBJECTIVE:Systematically review and appraise evaluation tools for patient and public engagement in research and health system decision making. METHODS:We searched literature published between January 1980 and February 2016. Electronic databases (Ovid MEDLINE, Embase, Cochrane Database of Systematic Reviews, CINAHL and PsycINFO) were consulted, as well as grey literature obtained through Google, subject-matter experts, social media and engagement organization websites. Two independent reviewers appraised the evaluation tools based on 4 assessment criteria: scientific rigour, patient and public perspective, comprehensiveness and usability. RESULTS:In total, 10 663 unique references were identified, 27 were included. Most of these tools were developed in the last decade and were designed to support improvement of engagement activities. Only 11% of tools were explicitly based on a literature review, and just 7% were tested for reliability. Patients and members of the public were involved in designing 56% of the tools, mainly in the piloting stage, and 18.5% of tools were designed to report evaluation results to patients and the public. CONCLUSION:A growing number of evaluation tools are available to support patient and public engagement in research and health system decision making. However, the scientific rigour with which such evaluation tools are developed could be improved, as well as the level of patient and public engagement in their design and reporting.
Project description:BACKGROUND:Low-value care initiatives are rapidly growing; however, it is not clear how members of the public should be involved. The objective of this scoping review was to systematically examine the literature describing public involvement in initatives to reduce low-value care. METHODS:Evidence sources included MEDLINE, EMBASE, and CINAHL databases from inception to November 26, 2019, grey literature (CADTH Tool), reference lists of included articles, and expert consultation. Citations were screened in duplicate and included if they referred to the public's perception and/or involvement in reducing low-value care. Public included patients or citizens without any advanced healthcare knowledge. Low-value care included medical tests or treatments that lack efficacy, have risks that exceed benefit, or are not cost-effective. Extracted data pertained to study characteristics, low-value practice, clinical setting, and level of public involvement (i.e., patient-clinician interaction, research, or policy-making). RESULTS:The 218 included citations were predominantly original research (n = 138, 63%), published since 2010 (n = 192, 88%), originating from North America (n = 146, 67%). Most citations focused on patient engagement within the patient-clinician interaction (n = 156, 72%), using tools that included shared decision-making (n = 66, 42%) and patient-targeted educational materials (n = 72, 46%), and reported both reductions in low-value care and improved patient perceptions regarding low-value care. Fewer citations examined public involvement in low-value care policy-making (n = 33, 15%). Among citations that examined perspectives regarding public involvement in initiatives to reduce low-value care (n = 10, 5%), there was consistent support for the utility of tools applied within the patient-clinician interaction and less consistent support for involvement in policy-making. CONCLUSIONS:Efforts examining public involvement in low-value care concentrate within the patient-clinician interaction, wherein patient-oriented educational materials and shared decision-making tools have been commonly studied and are associated with reductions in low-value care. This contrasts with inclusion of the public in low-value care policy decisions wherein tools to promote engagement are less well-developed and involvement not consistently viewed as valuable. TRIAL REGISTRATION:Open Science Framework (https://osf.io/6fsxm).
Project description:<h4>Background</h4>Patient, public, consumer, and community (P2C2) engagement in organization-, community-, and systemlevel healthcare decision-making is increasing globally, but its formal evaluation remains challenging. To define a taxonomy of possible P2C2 engagement metrics and compare existing evaluation tools against this taxonomy, we conducted a systematic review.<h4>Methods</h4>A broad search strategy was developed for English language publications available from January 1962 through April 2015 in PubMed, Embase, Sociological Abstracts, PsycINFO, EconLit, and the gray literature. A publication was excluded if: (1) the setting was not healthcare delivery (ie, we excluded non-health sectors, such as urban planning; research settings; and public health settings not involving clinical care delivery); (2) the P2C2 engagement was episodic; or (3) the concept of evaluation or possible evaluation metrics were absent. To be included as an evaluation tool, publications had to contain an evaluative instrument that could be employed with minimal modification by a healthcare organization.<h4>Results</h4>A total of 199 out of 3953 publications met exclusion and inclusion criteria. These were qualitatively analyzed using inductive content analysis to create a comprehensive taxonomy of 116 possible metrics for evaluating P2C2 engagement. 44 outcome metrics were grouped into three domains (internal, external, and aggregate outcomes) that included six subdomains: impact on engagement participants, impact on services provided by the healthcare organization, impact on the organization itself, influence on the broader public, influence on population health, and engagement cost-effectiveness. The 72 process metrics formed four domains (direct process metrics; surrogate process metrics; aggregate process metrics; and preconditions for engagement) that comprised sixteen subdomains. We identified 23 potential tools for evaluating P2C2 engagement. The identified tools were published between 1973-2015 and varied in their coverage of the taxonomy, methodology used (qualitative, quantitative, or mixed), and intended evaluators (organizational leaders, P2C2 participants, external evaluators, or some combination). Parts of the metric taxonomy were absent from all tools.<h4>Conclusion</h4>By comprehensively mapping potential outcome and process metrics as well as existing P2C2 engagement tools, this review supports high-quality P2C2 engagement globally by informing the selection of existing evaluation tools and identifying gaps where new tools are needed.
Project description:BACKGROUND:Patient and public engagement in paediatric health-care decision making is under-researched, and there is a lack of systematically reviewed literature in this area. OBJECTIVE:To examine the extent, range and nature of published research investigating the engagement of children/youth, families and the public in paediatric service improvement, to summarize key aspects of the research identified and to identify gaps to help inform future research needs. METHODS:Literature was sought in MEDLINE, EMBASE, PsycINFO and CINAHL. Eligible articles presented research focused on patient, family and public engagement strategies in the paediatric health-care setting. Two reviewers extracted and charted data and analysed findings using a descriptive numerical summary analysis and a thematic analysis. RESULTS:From 4331 articles, 21 were eligible. Most were from the United States. The majority of studies were undertaken in hospital settings and used quantitative methods. Various patient and public engagement strategies/interventions were examined, including shared decision-making tools, questionnaires, youth councils/family advisory groups, patient portals and online networks. Most of the studies examined child/youth/parent satisfaction, with fewer investigating treatment outcomes or service improvement. The majority of studies investigated an engagement strategy at the 'individual treatment' level of engagement. Regarding the continuum of engagement, most of the studies were at either the 'consultation' or 'involvement' stage. CONCLUSION:Future research needs to focus on the investigation of engagement strategies delivered in primary care, and the use of more qualitative and mixed methods approaches is recommended. There is a gap in the area of engagement strategies directed towards 'service design and resources' and 'macro/policy' levels.
Project description:Patient and public involvement and engagement (PPIE) in systematic reviews remains uncommon, despite the policy imperative for patient involvement in research. The aim of this study was to investigate the process and impact of collaborating with members of a patient Research User Group (RUG) on a systematic review about shared decision making around prescribing analgesia in primary care consultations.Five members of an established patient RUG collaborated with researchers undertaking a systematic review with narrative synthesis, through workshops held at three time-points. These addressed the following: designing the protocol, interpreting the results and planning dissemination. Support from a RUG coordinator and user support worker facilitated collaboration throughout the review process. Researchers reflected on how PPIE modified the review at each time-point.RUG members identified factors important in shared decision making around analgesic prescribing additional to those initially proposed by the research team. Search terms and specific outcomes of interest were amended to reflect these additional factors. Thirty of the 39 patient-identified factors were absent in the published literature. The categories of factors identified were used as a framework for the narrative synthesis and for reporting results. RUG members prioritized options for disseminating the results.PPIE collaboration throughout the systematic review impacted on the scope of the review, highlighting gaps in the literature that were important to patients. Impact on interpretation and dissemination of findings ensured the review directly reflected patient priorities. Challenges and strategies to facilitate PPIE involvement in systematic reviews and suggestions for future researchers are highlighted.
Project description:<b>Objective:</b> Patients with cystic fibrosis (CF) undertake time-consuming programs of home therapies. Our objective was to develop a tool to help CF patients prioritize personal goals for some of these treatments. We describe the development and results of initial evaluation of this shared decision-making tool. <b>Methods:</b> Multicriteria decision-making method to develop a shared decision-making tool that integrates patient's values and perceptions of treatment impact on functionality/sense of well-being. Treatment efficacy data obtained through comprehensive review of English language literature and Cochrane reviews. Field study of 21 patients was performed to assess acceptability of the approach, understandability of the tool, and to determine whether there was sufficient patient-to-patient variability in treatment goals and patient preferences to make use of a personalized tool worthwhile. <b>Results:</b> Patients found the tool easy to understand and felt engaged as active participants in their care. The tool was responsive to variations in patient preferences. Priority scores were calculated (0-1.0 ± SD). Patients' most important treatment goals for improving lung health included improving breathing function (0.27 ± 0.11), improving functionality/sense of well-being (0.24 ± 0.13), preventing lung infection (0.21 ± 0.08), minimizing time to complete treatments (0.16 ± 0.12), and minimizing cost (0.11 ± 0.09). <b>Conclusions:</b> A shared decision-making tool that integrates patients' values and best evidence is feasible and could result in improved patient engagement in their own care.
Project description:Public health decision-making structures in England have transformed since the implementation of reforms in 2013, with responsibility for public health services and planning having shifted from the "health" boundary to local authority (LA; local government) control. This transformation may have interrupted flows of research evidence use in decision-making and introduced a new political element to public health decision-making. For generators of research evidence, understanding and responding to this new landscape and decision-makers' evidence needs is essential.We conducted a systematic scoping review of the literature, drawing upon four databases and undertaking manual searching and citation tracking. Included studies were English-based, published in 2010 onwards, and were focused on public health decision-making, including the utilisation or underutilisation of research evidence use, in local (regional or sub-regional) areas. All studies presented empirical findings collected through primary research methods or through the reanalysis of existing primary data.From a total of 903 records, 23 papers from 21 studies were deemed to be eligible and were included for further data extraction. Three clear trends in evidence use were identified: (i) the primacy of local evidence, (ii) the important role of local experts in providing evidence and knowledge, and (iii) the high value placed on local evaluation evidence despite the varying methodological rigour. Barriers to the use of research evidence included issues around access and availability of applicable research evidence, and indications that the use of evidence could be perceived as a bureaucratic process. Two new factors resulting from reforms to public health structures were identified that potentially changed existing patterns of research evidence use and decision-making requirements: (i) greater emphasis among public health practitioners on the perceived uniqueness of LA areas and structures following devolution of public health into LAs and (ii) challenges introduced in responding to higher levels of local political accountability.There is a need to better understand and respond to the evidence needs of decision-makers working in public health and to work more collaboratively in developing solutions to the underutilisation of research evidence in decision-making.
Project description:BACKGROUND:Showing how engagement adds value for all stakeholders can be an effective motivator for broader implementation of patient engagement. However, it is unclear what methods can best be used to evaluate patient engagement. This paper is focused on ways to evaluate patient engagement at three decision-making points in the medicines research and development process: research priority setting, clinical trial design and early dialogues with regulators and health technology assessment bodies. OBJECTIVE:Our aim was to review the literature on monitoring and evaluation of patient engagement, with a focus on indicators and methods. SEARCH STRATEGY AND INCLUSION CRITERIA:We undertook a scoping literature review using a systematic search, including academic and grey literature with a focus on evaluation approaches or outcomes associated with patient engagement. No date limits were applied other than a cut-off of publications after July 2018. DATA EXTRACTION AND SYNTHESIS:Data were extracted from 91 publications, coded and thematically analysed. MAIN RESULTS:A total of 18 benefits and 5 costs of patient engagement were identified, mapped with 28 possible indicators for their evaluation. Several quantitative and qualitative methods were found for the evaluation of benefits and costs of patient engagement. DISCUSSION AND CONCLUSIONS:Currently available indicators and methods are of some use in measuring impact but are not sufficient to understand the pathway to impact, nor whether interaction between researchers and patients leads to change. We suggest that the impacts of patient engagement can best be determined not by applying single indicators, but a coherent set of measures.
Project description:Current research suggests that while patients are becoming more engaged across the health delivery spectrum, this involvement occurs most often at the pre-preparation stage to identify 'high-level' priorities in health ecosystem priority setting, and at the preparation phase for health research.The purpose of this systematic rapid review of the literature is to describe the evidence that does exist in relation to patient and public engagement priority setting in both health ecosystem and health research.HealthStar (via OVID); CINAHL; Proquest Databases; and Scholar's Portal.i) published in English; ii) published within the timeframe of 2007-Current (10 years) unless the report/article was formative in synthesizing key considerations of patient engagement in health ecosystem and health research priority setting; iii) conducted in Canada, the US, Europe, UK, Australia/New Zealand, or Scandinavian countries.i) Is the research valid, sound, and applicable?; ii) what outcomes can we potentially expect if we implement the findings from this research?; iii) will the target population (i.e., health researchers and practitioners) be able to use this research?. A summary of findings from each of the respective processes was synthesized to highlight key information that would support decision-making for researchers when determining the best priority setting process to apply for their specific patient-oriented research.Seventy articles from the UK, US, Canada, Netherlands and Australia were selected for review. Results were organized into two tiers of public and patient engagement in prioritization: Tier 1-Deliberative and Tier 2-Consultative. Highly structured patient and public engagement planning activities include the James Lind Alliance Priority Setting Partnerships (UK), Dialogue Method (Netherlands), Global Evidence Mapping (Australia), and the Deep Inclusion Method/CHoosing All Together (US).The critical study limitations include challenges in comprehensively identifying the patient engagement literature for review, bias in article selection due to the identified scope, missed information due to a more limited use of exhaustive search strategies (e.g., in-depth hand searching), and the heterogeneity of reported study findings.The four public and patient engagement priority setting processes identified were successful in setting priorities that are inclusive and objectively based, specific to the priorities of stakeholders engaged in the process. The processes were robust, strategic and aimed to promote equity in patient voices. Key limitations identified a lack of evaluation data on the success and extent in which patients were engaged. Issues pertaining to feasibility of stakeholder engagement, coordination, communication and limited resources were also considered.
Project description:BACKGROUND:Patient-Powered Research Networks (PPRNs) are a unique type of patient-powered patient registry for patient-centered outcomes research requiring that stakeholder engagement play a key role in governance (eg, research guidance and decision making). The purpose of this report is to describe the governance structure of a newly formed PPRN and the activities undertaken prelaunch and postlaunch to evaluate and improve the engagement of patient stakeholders in governance. METHODS:During the 18-month start-up period of ArthritisPower, a PPRN for adult rheumatologic conditions, 12 members of the patient body of the PPRN governance completed a 12-item preassessment and postassessment based on the Patient-Centered Outcomes Research Institute (PCORI) Engagement Rubric. The assessment was intended to measure Patient Governor (PG) perceptions of their engagement in governance within the first 3 months of their involvement at an in-person orientation. Six months later, the PG Chair initiated a mid-year evaluation with the same group to identify areas for improvement. Semistructured phone interviews were conducted with 11 PGs who were asked to rate and explain their perceptions about their participation in PPRN governance, the progress of the PPRN toward conducting research, the support they receive from staff, and the support they receive from other PGs. Results were compiled and interpreted by the Chair with help from the coprincipal investigator. RESULTS:Preassessment/postassessments indicated that PGs understood their governance role and decision-making authority after in-person orientation and felt that major PPRN decisions were being made with their input. Feedback and scores from the PG-led mid-year evaluation coalesced around 3 themes: a preference for receiving news and updates via email to allow more discussion and decision making during conference calls, a desire for guidance about how best to help advance the PPRN toward the conduct of research, and a need to communicate with each other as a group outside of monthly conference calls. Suggested activities to support patient engagement in PPRN governance include communicating clear expectations, providing well-prepared tools for engagement, and conducting regular assessments. CONCLUSIONS:Members of an online patient community are willing to share their expertise to participate in and shape research governance and bring both their professional and lived health experience to the development and improvement of PPRN governance structure. A patient-initiated and patient-led evaluation of governance communication procedures within the PPRN provided more specific recommendations for improvement than did an investigator-led preevaluation/postevaluation based on the PCORI Engagement Rubric.
Project description:Background:Decision-making in public health and health policy is complex and requires careful deliberation of many and sometimes conflicting normative and technical criteria. Several approaches and tools, such as multi-criteria decision analysis, health technology assessments and evidence-to-decision (EtD) frameworks, have been proposed to guide decision-makers in selecting the criteria most relevant and appropriate for a transparent decision-making process. This study forms part of the development of the WHO-INTEGRATE EtD framework, a framework rooted in global health norms and values as reflected in key documents of the World Health Organization and the United Nations system. The objective of this study was to provide a comprehensive overview of criteria used in or proposed for real-world decision-making processes, including guideline development, health technology assessment, resource allocation and others. Methods:We conducted an overview of systematic reviews through a combination of systematic literature searches and extensive reference searches. Systematic reviews reporting criteria used for real-world health decision-making by governmental or non-governmental organization on a supranational, national, or programme level were included and their quality assessed through a bespoke critical appraisal tool. The criteria reported in the reviews were extracted, de-duplicated and sorted into first-level (i.e. criteria), second-level (i.e. sub-criteria) and third-level (i.e. decision aspects) categories. First-level categories were developed a priori using a normative approach; second- and third-level categories were developed inductively. Results:We included 36 systematic reviews providing criteria, of which one met all and another eleven met at least five of the items of our critical appraisal tool. The criteria were subsumed into 8 criteria, 45 sub-criteria and 200 decision aspects. The first-level of the category system comprised the following seven substantive criteria: "Health-related balance of benefits and harms"; "Human and individual rights"; "Acceptability considerations"; "Societal considerations"; "Considerations of equity, equality and fairness"; "Cost and financial considerations"; and "Feasibility and health system considerations". In addition, we identified an eight criterion "Evidence". Conclusion:This overview of systematic reviews provides a comprehensive overview of criteria used or suggested for real-world health decision-making. It also discusses key challenges in the selection of the most appropriate criteria and in seeking to implement a fair decision-making process.