The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research.
ABSTRACT: The introduction of evidence-based programs and practices into healthcare settings has been the subject of an increasing amount of research in recent years. While a number of studies have examined initial implementation efforts, less research has been conducted to determine what happens beyond that point. There is increasing recognition that the extent to which new programs are sustained is influenced by many different factors and that more needs to be known about just what these factors are and how they interact. To understand the current state of the research literature on sustainability, our team took stock of what is currently known in this area and identified areas in which further research would be particularly helpful. This paper reviews the methods that have been used, the types of outcomes that have been measured and reported, findings from studies that reported long-term implementation outcomes, and factors that have been identified as potential influences on the sustained use of new practices, programs, or interventions. We conclude with recommendations and considerations for future research.Two coders identified 125 studies on sustainability that met eligibility criteria. An initial coding scheme was developed based on constructs identified in previous literature on implementation. Additional codes were generated deductively. Related constructs among factors were identified by consensus and collapsed under the general categories. Studies that described the extent to which programs or innovations were sustained were also categorized and summarized.Although "sustainability" was the term most commonly used in the literature to refer to what happened after initial implementation, not all the studies that were reviewed actually presented working definitions of the term. Most study designs were retrospective and naturalistic. Approximately half of the studies relied on self-reports to assess sustainability or elements that influence sustainability. Approximately half employed quantitative methodologies, and the remainder employed qualitative or mixed methodologies. Few studies that investigated sustainability outcomes employed rigorous methods of evaluation (e.g., objective evaluation, judgement of implementation quality or fidelity). Among those that did, a small number reported full sustainment or high fidelity. Very little research has examined the extent, nature, or impact of adaptations to the interventions or programs once implemented. Influences on sustainability included organizational context, capacity, processes, and factors related to the new program or practice themselves.Clearer definitions and research that is guided by the conceptual literature on sustainability are critical to the development of the research in the area. Further efforts to characterize the phenomenon and the factors that influence it will enhance the quality of future research. Careful consideration must also be given to interactions among influences at multiple levels, as well as issues such as fidelity, modification, and changes in implementation over time. While prospective and experimental designs are needed, there is also an important role for qualitative research in efforts to understand the phenomenon, refine hypotheses, and develop strategies to promote sustainment.
Project description:BACKGROUND:Implementation scientists and practitioners, alike, recognize the importance of sustaining practice change, however post-implementation studies of interventions are rare. This is a protocol for the Sustainment, Sustainability and Spread Study (SSaSSy). The purpose of this study is to contribute to knowledge on the sustainment (sustained use), sustainability (sustained benefits), and spread of evidence-based practice innovations in health care. Specifically, this is a post-implementation study of an evidence-informed, Care Aide-led, facilitation-based quality-improvement intervention called SCOPE (Safer Care for Older Persons (in long-term care) Environments). SCOPE has been implemented in nursing homes in the Canadian Provinces of Manitoba (MB), Alberta (AB) and British Columbia (BC). Our study has three aims: (i) to determine the role that adaptation/contextualization plays in sustainment, sustainability and spread of the SCOPE intervention; (ii) to study the relative effects on sustainment, sustainability and intra-organizational spread of high-intensity and low-intensity post-implementation "boosters", and a "no booster" condition, and (iii) to compare the relative costs and impacts of each booster condition. METHODS/DESIGN:SSaSSy is a two-phase mixed methods study. The overarching design is convergent, with qualitative and quantitative data collected over a similar timeframe in each of the two phases, analyzed independently, then merged for analysis and interpretation. Phase 1 is a pilot involving up to 7 units in 7 MB nursing homes in which SCOPE was piloted in 2016 to 2017, in preparation for phase 2. Phase 2 will comprise a quasi-experiment with two treatment groups of low- and high-intensity post-implementation "boosters", and an untreated control group (no booster), using pretests and post-tests of the dependent variables relating to sustained care and management practices, and resident outcomes. Phase 2 will involve 31 trial sites in BC (17 units) and AB (14 units) nursing homes, where the SCOPE trial concluded in May 2019. DISCUSSION:This project stands to advance understanding of the factors that influence the sustainment of practice changes introduced through evidence-informed practice change interventions, and their associated sustainability. Findings will inform our understanding of the nature of the relationship of fidelity and adaptation to sustainment and sustainability, and afford insights into factors that influence the intra-organizational spread of practice changes introduced through complex interventions.
Project description:BACKGROUND:Sustainability frameworks differentiate between sustainability capacity and sustainment of organizational change. Multiple studies have examined sustainability capacity. Methodological approaches to assess long-term sustainment have not been explored. This study addresses this gap by describing the development of a long-term sustainment methodology and evaluating its application utilizing data from substance abuse clinics participating in a quality improvement collaborative. METHODS:The study involved clinics (n?=?121) in three states (MI, NY and WA) participating in the 2007-2009 NIATx200 quality improvement (QI) intervention. It extended the primary analysis to focus on clinics' long-term sustainment of wait time, retention and admission improvements. Long-term sustainment was defined as two years post end of the active implementation period (Calendar Years 2010 and 2011). The analysis defined case exclusion criteria and spline "knot" time intervals; allowed for Cp statistic use to address clinic data volatility; established the structure of sustainment plots and explored differences between NIATx implementation strategies. RESULTS:Example spline and sustain plots highlight the application of the long-term sustainment methodology for NIATx200 clinics. In clinics with available longitudinal outcome data, 40.8% (n?=?31 of 76 clinics) sustained improvements in wait time, 26.7% (n?=?20 of 75 clinics) in retention, and 28.1% (n?=?32 of 114 clinics) for admissions. Clinic assignment to a NIATx200 implementation strategy did not significantly influence a clinics' long-term sustainment except for lower wait time changes in the interest circle interventions. Thirty clinics (24.8%) sustained improvements for two outcomes and six clinics (5.0%) did so for all three outcomes. The clinics that sustained multiple outcome improvements were assigned to the interest circle (n?=?12), learning session (n?=?10), combination (n?=?8), and coaching (n?=?6) implementation strategies. Guidance for applying the long-term sustainment methodology in other quality improvement settings is described. CONCLUSIONS:Research about sustainability capacity and sustainment of change has become increasingly important in dissemination and implementation research. Assessment of long-term sustainment in a multi-organizational quality improvement collaborative (QIC) is needed to identify when program drift and intervention decay occurs. If "cut-points" indicate when effects diminish, specific sustainability modules could be developed and introduced within the structure of a QIC to improve organizational long-term sustainment. Coaches and change teams could be trained to focus on organizational change sustainment and strengthen the likelihood of institutionalization. ClinicalTrials.gov Identifier: NCT00934141 Registered July 6, 2009. Retrospectively registered.
Project description:Background:One goal of health systems seeking to evolve into learning health systems is to accelerate the implementation and sustainment of evidence-based practices (EBPs). As part of this evolution, the Veterans Health Administration (VHA) developed the Innovation Ecosystem, which includes the Diffusion of Excellence (DoE), a program that identifies and diffuses Gold Status Practices (GSPs) across facilities. The DoE hosts an annual "Shark Tank" competition in which leaders bid on the opportunity to implement a GSP with 6 months of implementation support. Over 750 diverse practices were submitted in cohorts 2 and 3 of Shark Tank; 23 were designated GSPs and were implemented in 31 VA networks or facilities. As part of a national evaluation of the DoE, we identified factors contributing to GSP implementation and sustainment. Methods:Our sequential mixed methods evaluation of cohorts 2 and 3 of Shark Tank included semi-structured interviews with at least one representative from 30/31 implementing teams (N = 78/105 people invited) and survey responses from 29/31 teams (N = 39/47 invited). Interviews focused on factors influencing implementation and future sustainment. Surveys focused on sustainment 1.5-2?years after implementation. The Consolidated Framework for Implementation Research (CFIR) informed data collection and directed content analysis. Ordinal scales were developed inductively to rank implementation and sustainment outcomes. Results:Over 50% of teams (17/30) successfully implemented their GSP within the 6-month implementation period. Despite extensive implementation support, significant barriers related to centralized decision-making, staffing, and resources led to partial (n = 6) or no (n = 7) implementation for the remaining teams. While 12/17 initially successful implementation teams reported sustained use of their GSP, over half of the initially unsuccessful teams (n = 7/13) also reported sustained GSP use 1.5?years after the initial implementation period. When asked at 6 months, 18/27 teams with complete data accurately anticipated their future sustainability based on reported sustainment an average of 1.5?years later. Conclusions:Most teams implemented within 6 months and/or sustained their GSP 1.5?years later. High levels of implementation and sustainment across diverse practices and teams suggest that VHA's DoE is a successful large-scale model of diffusion. Team predictions about sustainability after the first 6 months of implementation provide a promising early assessment and point of intervention to increase sustainability.
Project description:BACKGROUND:In 2012, the Saskatchewan Ministry for Health mandated a system-wide Lean transformation. Research has been conducted on the implementation processes of this system-wide Lean implementation. However, no research has been done on the sustainability of these Lean efforts. We conducted a realist evaluation on the sustainability of Lean in pediatric healthcare. We used the context (C)?+?mechanism (M)?=?outcome (O) configurations (CMOcs) heuristic to explain under what contexts, for whom, how and why Lean efforts are sustained or not sustained in pediatric healthcare. METHODS:We employed a case study research design. Guided by a realist evaluation framework, we conducted qualitative realist interviews with various stakeholder groups across four pediatric hospital units 'cases' at one acute hospital. Interview data was analyzed using an integrated approach of CMOc categorization coding, CMOc connecting and pattern matching. RESULTS:We conducted thirty-two interviews across the four cases. Five CMOcs emerged from our realist interview data. These configurations illustrated a 'ripple-effect' from implementation outcomes to contexts for sustainability. Sense-making and staff engagement were prominent mechanisms to the sustainment of Lean efforts. Failure to trigger these mechanisms resulted in resistance. The implementation approach used influenced mechanisms and outcomes for sustainability, more so than Lean itself. Specifically, the language, messaging and training approaches used triggered mechanisms of innovation fatigue, poor 'sense-making' and a lack of engagement for frontline staff. The mandated, top-down, externally led nature of implementation and lack of customization to context served as potential pitfalls. Overall, there was variation between leadership and frontline staff's perceptions on how embedded Lean was in their contexts, and the degree to which participants supported Lean sustainability. CONCLUSIONS:This research illuminates important contextual factors and mechanisms to the process of Lean sustainment that can be applicable to those implementing systems changes. Future work is needed to continue to develop the science on the sustainability of interventions for healthcare improvement.
Project description:Implementation processes are often long and complex, requiring sustained facilitation efforts. Drawing on organizational and implementation literature, we examined the influence of senior management support (SMS), middle management support (MMS), facilitator team time availability (TIME) and team continuity (CONTINUITY) on sustainment of internal facilitation activities. For 2 years, we followed 10 small rural hospitals implementing TeamSTEPPS, a patient safety program, and conducted quarterly interviews with key informants. We coded, calibrated, and analyzed the data using the fuzzy-set qualitative comparative analysis. We found that five hospitals sustained facilitation activities and the combination of SMS, MMS, and CONTINUITY (i.e., presence of all three factors) was a sufficient condition for sustainment. Five other hospitals did not sustain facilitation activities and they either lacked MMS or lacked both TIME and CONTINUITY. In follow-up analyses, we found that team leadership continuity also influenced sustainment patterns. We discussed the implications for research and practice.
Project description:Research to date has largely focused on predictors of adoption and initial implementation of evidence-based practices (EBPs), yet sustained implementation is crucial to deliver a return on investments in dissemination. Furthermore, most studies focus on single EBPs, limiting opportunities to study the fit between practice characteristics EBPs and implementation contexts.This observational study will characterize implementation sustainment and identify organizational and therapist characteristics that predict sustainment of multiple practices being implemented within a fiscal mandate in the largest public mental health system in the USA. Specific aims are to (1) characterize sustainment outcomes (volume/penetration, EBP concordant care); (2) use mixed methods to characterize inner context (agency- and therapist-level) factors and early implementation conditions; and (3) identify inner context factors and early implementation conditions that predict sustainment outcomes. This study will undertake original data collection and analysis of existing data sources to achieve its aims. Archived reports and documents will be used to characterize early implementation conditions in 102 agencies. Administrative claims data will be used to characterize volume and penetration outcomes over 8 years. Therapist and program manager surveys will be administered to characterize sustained EBP concordant care and inner context determinants of sustainment. An in-depth study in a subset of agencies will yield interview data and recordings of treatment sessions for validation of the EBP concordant care scale.This project will yield new understanding of whether and how multiple EBPs can be sustained in public mental health systems undergoing a policy-driven community implementation effort. We will produce generalizable models for characterizing sustainment, including feasible and flexible measurement of practice across multiple EBPs. The findings will inform the development of implementation interventions to promote sustained delivery of EBPs to maximize their public health impact.
Project description:Sustaining prevention efforts directed at substance use and mental health problems is one of the greatest, yet least understood, challenges in the field of implementation science. A large knowledge gap exists regarding the meaning of the term "sustainment" and what factors predict or even measure sustainability of effective prevention programs and support systems.The U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) supports a diverse portfolio of prevention and treatment grant programs that aim to improve population and individual level behavioral health. This study focuses on four SAMHSA prevention grant programs, two of which target substance abuse prevention at the state or single community level, one targets suicide prevention, and one targets prevention of aggressive/disruptive behavior in elementary schools. An examination of all four grant programs simultaneously provides an opportunity to determine what is meant by the term sustainment and identify and support both the unique requirements for improving sustainability for each program as well as for developing a generalizable framework comprised of core components of sustainment across diverse prevention approaches. Based on an analysis of qualitative and quantitative data of 10 grantees supported by these four programs, we will develop a flexible measurement system, with both general and specific components, that can bring precision to monitoring sustainment of infrastructure, activities, and outcomes for each prevention approach. We will then transform this system for use in evaluating and improving the likelihood of achieving prevention effort sustainment. To achieve these goals, we will (1) identify core components of sustainment of prevention programs and their support infrastructures; (2) design a measurement system for monitoring and providing feedback regarding sustainment within the four SAMHSA's prevention-related grant programs; and (3) pilot test the predictability of this multilevel measurement system across these programs and the feasibility and acceptability of a measurement system to evaluate and improve the likelihood of sustainment.This project is intended to improve sustainment of the supporting prevention infrastructure, activities, and outcomes that are funded by federal, state, community, and foundation sources.
Project description:BACKGROUND:Large-scale implementation of evidence-based psychotherapies (EBPs) such as cognitive processing therapy (CPT) for posttraumatic stress disorder can have a tremendous impact on mental and physical health, healthcare utilization, and quality of life. While many mental health systems (MHS) have invested heavily in programs to implement EBPs, few eligible patients receive EBPs in routine care settings, and clinicians do not appear to deliver the full treatment protocol to many of their patients. Emerging evidence suggests that when CPT and other EBPs are delivered at low levels of fidelity, clinical outcomes are negatively impacted. Thus, identifying strategies to improve and sustain the delivery of CPT and other EBPs is critical. Existing literature has suggested two competing strategies to promote sustainability. One emphasizes fidelity to the treatment protocol through ongoing consultation and fidelity monitoring. The other focuses on improving the fit and effectiveness of these treatments through appropriate adaptations to the treatment or the clinical setting through a process of data-driven, continuous quality improvement. Neither has been evaluated in terms of impact on sustained implementation. METHODS:To compare these approaches on the key sustainability outcomes and provide initial guidance on sustainability strategies, we propose a cluster randomized trial with mental health clinics (n?=?32) in three diverse MHSs that have implemented CPT. Cohorts of clinicians and clinical managers will participate in 1 year of a fidelity oriented learning collaborative or 1 year of a continuous quality improvement-oriented learning collaborative. Patient-level PTSD symptom change, CPT fidelity and adaptation, penetration, and clinics' capacity to deliver EBP will be examined. Survey and interview data will also be collected to investigate multilevel influences on the success of the two learning collaborative strategies. This research will be conducted by a team of investigators with expertise in CPT implementation, mixed method research strategies, quality improvement, and implementation science, with input from stakeholders in each participating MHS. DISCUSSION:It will have broad implications for supporting ongoing delivery of EBPs in mental health and healthcare systems and settings. The resulting products have the potential to significantly improve efforts to ensure ongoing high quality implementation and consumer access to EBPs. TRIAL REGISTRATION:NCT02449421 . Registered 02/09/2015.
Project description:BACKGROUND:Sustained implementation of school-based prevention programs is low. Effective strategies are needed to enhance both high-level implementation fidelity and sustainability of prevention programs. OBJECTIVE:This proposed study aims to determine if the provision of either biweekly monitoring and feedback and site-based assistance and mentorship or both to at-risk and moderate-performing teachers with monitoring through an enhanced decision-making platform by the Ministry of Education (MOE) and Ministry of Health (MOH) based on the real-time implementation data will increase national implementation fidelity and result in sustained implementation over time. METHODS:This study will target government schools including 200 grade 6 teachers in 80 primary schools and 100 junior/middle high school teachers (and their classes) on 12 Bahamian islands. Teacher and school coordinator training will be conducted by the MOE in year 1, followed by an optimization trial among teachers in the capital island. Informed by these results, an implementation intervention will be conducted to train using different levels of educational intensity all at-risk and moderate-performing teachers. Subsequently selected training and implementation strategies will be evaluated for the national implementation of Focus on Youth in the Caribbean and Caribbean Informed Parents and Children Together in years 2 to 5. RESULTS:It is hypothesized that a more intensive training and supervision program for at-risk and moderate-performing teachers will enhance their implementation fidelity to the average level of the high-performing group (85%), an HIV prevention program delivered at the national level can be implemented with fidelity in grade 6 and sustained over time (monitored annually), and student outcomes will continue to be highly correlated with implementation fidelity and be sustained over time (assessed annually through grade 9). The proposed study is funded by the National Institute of Child Health and Human Development from August 1, 2018, through May 31, 2023. CONCLUSIONS:The study will explore several theory-driven implementation strategies to increase sustained teacher implementation fidelity and thereby increase the general public health impact of evidence-based interventions. The proposed project has potential to make significant contributions to advancing school-based HIV prevention research and implementation science and serve as a global model for the Fast Track strategy. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID):PRR1-10.2196/14816.
Project description:Several studies have demonstrated clinical benefits of integrated care for a range of child and adolescent mental health outcomes. However, there is a significant gap between the evidence for efficacy of integrated care interventions vs their implementation in practice. While several studies have examined large-scale implementation of co-located integrated care for adults, much less is known for children. The goal of this scoping review was to understand how co-located mental health interventions targeting children and adolescents have been implemented and sustained. The literature was systematically searched for interventions targeting child and adolescent mental health that involved a mental health specialist co-located in a primary care setting. Studies reporting on the following implementation outcomes were included: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability. This search identified 34 unique studies, including randomized controlled trials, observational studies, and survey/mixed method approaches. Components facilitating implementation of on-site integrated behavioural healthcare included interprofessional communication and collaboration at all stages of implementation; clear protocols to facilitate intervention delivery; and co-employment of integrated care providers by specialty clinics. Some studies found differences in service use by demographic factors, and others reported funding challenges affecting sustainability, warranting further study.