Systematic evaluation of implementation fidelity of complex interventions in health and social care.
ABSTRACT: Evaluation of an implementation process and its fidelity can give insight into the 'black box' of interventions. However, a lack of standardized methods for studying fidelity and implementation process have been reported, which might be one reason for the fact that few prior studies in the field of health service research have systematically evaluated interventions' implementation processes.The aim of this project is to systematically evaluate implementation fidelity and possible factors influencing fidelity of complex interventions in health and social care.A modified version of The Conceptual Framework for Implementation Fidelity will be used as a conceptual model for the evaluation. The modification implies two additional moderating factors: context and recruitment. A systematic evaluation process was developed. Multiple case study method is used to investigate implementation of three complex health service interventions. Each case will be investigated in depth and longitudinally, using both quantitative and qualitative methods.This study is the first attempt to empirically test The Conceptual Framework for Implementation Fidelity. The study can highlight mechanism and factors of importance when implementing complex interventions. Especially the role of the moderating factors on implementation fidelity can be clarified.Supported Employment, SE, among people with severe mental illness -- a randomized controlled trial: NCT00960024.
Project description:BACKGROUND:Evaluation of complex interventions should include a process evaluation to give evaluators, researchers, and policy makers greater confidence in the outcomes reported from RCTs. Implementation fidelity can be part of a process evaluation and refers to the degree to which an intervention is delivered according to protocol. The aim of this implementation fidelity study was to evaluate to what extent a dialogue-based psychosocial intervention was delivered according to protocol. A modified conceptual framework for implementation fidelity was used to guide the analysis. METHODS:This study has an explanatory, sequential two-phase mixed methods design. Quantitative process data were collected longitudinally along with data collection in the RCT. Qualitative process data were collected after the last data collection point of the RCT. Descriptive statistical analyses were conducted to describe the sample, the intervention trajectories, and the adherence measures. A scoring system to clarify quantitative measurement of the levels of implementation was constructed. The qualitative data sources were analyzed separately with a theory-driven content analysis using categories of adherence and potential moderating factors identified in the conceptual framework of implementation fidelity. The quantitative adherence results were extended with the results from the qualitative analysis to assess which potential moderators may have influenced implementation fidelity and in what way. RESULTS:The results show that the core components of the intervention were delivered although the intervention trajectories were individualized. Based on the composite score of adherence, results show that 80.1% of the interventions in the RCT were implemented with high fidelity. Although it is challenging to assess the importance of each of the moderating factors in relation to the other factors and to their influence on the adherence measures, participant responsiveness, comprehensiveness of policy description, context, and recruitment appeared to be the most prominent moderating factors of implementation fidelity in this study. CONCLUSIONS:This evaluation of implementation fidelity and the discussion of what constitutes high fidelity implementation of this intervention are crucial in understanding the factors influencing the trial outcome. The study also highlights important methodological considerations for researchers planning process evaluations and studies of implementation fidelity. TRIAL REGISTRATION:ClinicalTrials.gov , NCT02338869; registered 10/04/2014.
Project description:Prior studies measuring fidelity of complex interventions have mainly evaluated adherence, and not taken factors affecting adherence into consideration. A need for studies that clarify the concept of fidelity and the function of factors moderating fidelity has been emphasized. The aim of the study was to systematically evaluate implementation fidelity and possible factors influencing fidelity of a complex care continuum intervention for frail elderly people.The intervention was a systematization of the collaboration between a nurse with geriatric expertise situated at the emergency department, the hospital ward staff, and a multi-professional team with a case manager in the municipal care services for older people. Implementation was evaluated between September 2008 and May 2010 with observations of work practices, stakeholder interviews, and document analysis according to a modified version of The Conceptual Framework for Implementation Fidelity.A total of 16 of the 18 intervention components were to a great extent delivered as planned, while some new components were added to the model. No changes in the frequency or duration of the 18 components were observed, but the dose of the added components varied over time. Changes in fidelity were caused in a complex, interrelated fashion by all the moderating factors in the framework, i.e., context, staff and participant responsiveness, facilitation, recruitment, and complexity.The Conceptual Framework for Implementation Fidelity was empirically useful and included comprehensive measures of factors affecting fidelity. Future studies should focus on developing the framework with regard to how to investigate relationships between the moderating factors and fidelity over time.ClinicalTrials.gov, NCT01260493.
Project description:Despite the implementation of various initiatives to address low enrollment in voluntary micro health insurance (MHI) schemes in sub-Saharan Africa, the problem of low enrollment remains unresolved. The lack of process evaluations of such interventions makes it difficult to ascertain whether their poor results are because of design failures or implementation weaknesses.In this paper, we describe a process evaluation protocol aimed at opening the 'black box' to evaluate the implementation processes of the Redesigned Community Health Fund (CHF) program in the Dodoma region of Tanzania.The study employs a cross-sectional mixed methods design and is being carried out 3 years after the launch of the Redesigned CHF program. The study is grounded in a conceptual framework which rests on the Diffusion of Innovation Theory and the Implementation Fidelity Framework. The study utilizes a mixture of quantitative and qualitative data collection tools (questionnaires, focus group discussions, in-depth interviews, and document review), and aligns the evaluation to the Theory of Intervention developed by our team. Quantitative data will be used to measure program adoption, implementation fidelity, and their moderating factors. Qualitative data will be used to explore the responses of stakeholders to the intervention, contextual factors, and moderators of adoption, implementation fidelity, and sustainability.This protocol describes a systematic process evaluation in relation to the implementation of a reformed MHI. We trust that the theoretical approaches and methodologies described in our protocol may be useful to inform the design of future process evaluations focused on the assessment of complex interventions, such as MHI schemes.
Project description:BACKGROUND:To cope with rising demand for healthcare services in Singapore, Regional Health Systems (RHS) comprising of health and social care providers across care settings were set up to integrate service delivery. Tasked with providing care for the western region, in 2012, the National University Health System (NUHS) - RHS developed a transitional care program for elderly patients with complex healthcare needs who consumed high levels of hospital resources. Through needs assessment, development of personalized care plans and care coordination, the program aimed to: (i) improve quality of care, (ii) reduce hospital utilization, and (iii) reduce healthcare-related costs. In this study, recognizing the need for process evaluation in conjunction with outcome evaluation, we aim to evaluate the implementation fidelity of the NUHS-RHS transitional care program to explain the outcomes of the program and to inform further development of (similar) programs. METHODS:Guided by the modified version of the Conceptual Framework for Implementation Fidelity (CFIF), adherence and moderating factors influencing implementation were assessed using non-participatory observations, reviews of medical records and program databases. RESULTS:Most (10 out of 14) components of the program were found to be implemented with low or moderate level of fidelity. The frequency or duration of the program components were observed to vary based on the needs of users, availability of care coordinators (CC) and their confidence. Variation in fidelity was influenced predominantly by: (1) complexity of the program, (2) extent of facilitation through guiding protocols, (3) facilitation of program implementation through CCs' level of training and confidence, (4) evolving healthcare participant responsiveness, and (5) the context of suboptimal capability among community providers. CONCLUSION:This is the first study to assess the context-specific implementation process of a transitional care program in the context of Southeast Asia. It provides important insights to facilitate further development and scaling up of transitional care programs within the NUHS-RHS and beyond. Our findings highlight the need for greater focus on engaging both healthcare providers and users, training CCs to equip them with the relevant skills required for their jobs, and building the capability of the community providers to implement such programs.
Project description:Background:The Tika Vaani intervention, an initiative to improve basic health knowledge and empower beneficiaries to improve vaccination uptake and child health for underserved rural populations in India, was assessed in a pilot cluster randomized trial. The intervention was delivered through two strategies: mHealth (using mobile phones to send vaccination reminders and audio-based messages) and community mobilization (face-to-face meetings) in rural Indian villages from January to September 2018. We assessed acceptability and implementation fidelity to determine whether the intervention delivered in the pilot trial can be implemented at a larger scale. Methods:We adapted the Conceptual Framework for implementation fidelity to assess acceptability and fidelity of the pilot interventions using a mixed methods design. Quantitative data sources include a structured checklist, household surveys, and mobile phone call patterns. Qualitative data came from field observations, intervention records, semi-structured interviews and focus groups with project recipients and implementers. Quantitative analyses assessed whether activities were implemented as planned, using descriptive statistics to describe participant characteristics and the percentage distribution of activities. Qualitative data were analyzed using content analysis and in the light of the implementation fidelity model to explore moderating factors and to determine how well the intervention was received. Results:Findings demonstrated high (86.7%) implementation fidelity. A total of 94% of the target population benefited from the intervention by participating in a face-to-face group meeting or via mobile phone. The participants felt that the strategies were useful means for obtaining information. The clarity of the intervention theory, the motivation, and commitment of the implementers as well as the periodic meetings of the supervisors largely explain the high level of fidelity obtained. Geographic distance, access to a mobile phone, level of education, and gender norms are contextual factors that contributed to heterogeneity in participation. Conclusions:Although the intervention was evaluated in the context of a randomized trial that could explain the high level of fidelity obtained, this evaluation provides confirmatory evidence that the results of the study reflect the underlying theory. The mobile platform coupled with community mobilization was well-received by the participants and could be a useful way to improve health knowledge and change behavior. Trial registration:ISRCTN 44840759 (22 April 2018).
Project description:International dissemination of evidence-based interventions calls for rigorous evaluation. As part of an evaluation of critical time intervention (CTI) for homeless people and abused women leaving Dutch shelters, this study assessed fidelity in two service delivery systems and explored factors influencing model adherence. Data collection entailed chart review (n = 70) and two focus groups with CTI workers (n = 11). The intervention obtained an overall score of three out of five (fairly implemented) for compliance fidelity and chart quality combined. Fidelity did not differ significantly between service systems, supporting its suitability for a range of populations. The eight themes that emerged from the focus groups as affecting model adherence provide guidance for future implementation efforts.
Project description:BACKGROUND:Accessibility to efficient and person-centered healthcare delivery drives healthcare transformation in many countries. In Singapore, specialist outpatient clinics (SOCs) are commonly congested due to increasing demands for chronic care. To improve this situation, the National University Health System (NUHS) Regional Health System (RHS) started an integrated care initiative,the Right-Site Care (RSC) program in 2014. Through collaborations between SOCs at the National University Hospital and primary and community care (PCC) clinics in the western region of the county, the program was designed to facilitate timely discharge and appropriate transition of patients, who no longer required specialist care, to the community. The aim of this study was to evaluate the implementation fidelity of the NUHS RHS RSC program using the modified Conceptual Framework for Implementation Fidelity (CFIF), at three distinct levels; providers, organizational, and system levels to explain outcomes of the program and to inform further development of (similar) programs. METHODS:A convergent parallel mixed methods study using the realist evaluation approach was used. Data were collected between 2016 and 2018 through non-participatory observations, reviews of medical records and program database, together with semi-structured interviews with healthcare providers. Triangulation of data streams was applied guided by the modified CFIF. RESULTS:Our findings showed four out of six program components were implemented with low level of fidelity, and 9112 suitable patients were referred to the program while 3032 (33.3%) declined to be enrolled. Moderating factors found to influence fidelity included: (i) complexity of program, (ii) evolving providers' responsiveness, (iii) facilitation through synergistic partnership, training of PCC providers by specialists and supportive structures: care coordinators, guiding protocols, shared electronic medical record and shared pharmacy, (iv) lack of organization reinforcement, and (v) mismatch between program goals, healthcare financing and providers' reimbursement. CONCLUSION:Functional integration alone is insufficient for a successful right-site care program implementation. Improvement in relationships between providers, organizations, and patients are also warranted for further development of the program.
Project description:This study investigated the utility of the Study of Health and Activity in Preschool Environments (SHAPES) conceptual model, which targeted physical activity (PA) behavior in preschool children, by examining the relationship between implementation monitoring data and child PA during the school day. We monitored implementation completeness and fidelity based on multiple elements identified in the conceptual model. Comparing high-implementing, low-implementing, and control groups revealed no association between implementation and outcomes. We performed post hoc analyses, using process data, to refine our conceptual model's depiction of an effective preschool PA-promoting environment. Results suggest that a single component of the original four-component conceptual model, providing opportunities for moderate-to-vigorous physical activity through recess for 4-year-old children in preschool settings, may be a good starting place for increasing moderate-to-vigorous physical activity. Interventions that are implemented with optimal levels of completeness and fidelity are more likely to achieve behavior change if they are based on accurate conceptual models. Examining the mechanisms through which an intervention produces its effects, as articulated in the conceptual model that guides it, is particularly important for environmentally focused interventions because they are guided by emerging frameworks. The results of this study underscore the utility of using implementation monitoring data to examine the conceptual model on which the intervention is based.
Project description:BACKGROUND:The shortage of organs for kidney transplantation for patients with end-stage renal disease (ESRD) is magnified in Hispanics/Latin Americans in the United States. Living donor kidney transplantation (LDKT) is the treatment of choice for ESRD. However, compared to their representation on the transplant waitlist, fewer Hispanics receive a LDKT than non-Hispanic whites. Barriers to LDKT for Hispanics include: lack of knowledge, cultural concerns, and language barriers. Few interventions have been designed to reduce LDKT disparities. This study aims to reduce Hispanic disparities in LDKT through a culturally targeted intervention. METHODS/DESIGN:Using a prospective effectiveness-implementation hybrid design involving pre-post intervention evaluation with matched controls, we will implement a complex culturally targeted intervention at two transplant centers in Dallas, TX and Phoenix, AZ. The goal of the study is to evaluate the effect of Northwestern Medicine's® Hispanic Kidney Transplant Program's (HKTP) key culturally targeted components (outreach, communication, education) on Hispanic LDKT rates over five years. The main hypothesis is that exposure to the HKTP will reduce disparities by increasing the ratio of Hispanic to non-Hispanic white LDKTs and the number of Hispanic LDKTs. We will also examine other process and outcome measures including: dialysis patient outreach, education session attendance, marketing efforts, Hispanic patients added to the waitlist, Hispanic potential donors per potential recipient, and satisfaction with culturally competent care. We will use mixed methods based on the Promoting Action on Research Implementation in Health Services (revised PARIHS) and the Consolidated Framework for Implementation Research (CFIR) frameworks to formatively evaluate the fidelity and innovative adaptations to HKTP's components at both study sites, to identify moderating factors that most affect implementation fidelity, and to identify adaptations that positively and negatively affect outcomes for patients. DISCUSSION:Our study will provide new knowledge about implementing culturally targeted interventions and their impact on reducing health disparities. Moreover, the study of a complex organizational-level intervention's implementation over five years is rare in implementation science; as such, this study is poised to contribute new knowledge to the factors influencing how organizational-level interventions are sustained over time. TRIAL REGISTRATION:(ClinicalTrials.gov registration # NCT03276390 , date of registration: 9-7-17, retrospectively registered).
Project description:Burkina Faso implemented the seasonal malaria chemoprevention (SMC) in 2014 in seven pilot health districts, following the new recommendation by the WHO in 2012 for the prevention of the disease in children under five years old, for areas of highly seasonal malaria transmission.The objective of this study was to assess the implementation fidelity of the seasonal malaria chemoprevention strategy in one of the districts, Kaya Health District.We conducted a case study, with a quantitative and qualitative mixed methods. Data were collected after two campaigns of implementation of the intervention, in 2014 and 2015, through a review of specific documents of SMC intervention, and individual interview with key informants (n = 21) involved at various levels in the implementation of the strategy and a household survey with the parents (n = 284) of eligible children for the SMC strategy in 2015 in the Kaya health district. The analysis framework focused on the fidelity of the intervention's content, its coverage, and its schedule, as well as the potential moderating factors, using the model proposed by Hasson, originally from Carroll.All components of the intervention were implemented. Villages and sectors were covered at 100%. In terms of intervention doses received, less than one-third of eligible children (32.3%) received the recommended four doses in 2015. Implementation of the strategy faced some difficulties due to insufficient training of community distributors, inadequate supply of inputs and insufficient financial resources for remuneration, advocacy and supervision, but also because of the contextual constraints due to the rainy season. Moreover, an interaction between the different moderating factors, influencing the degree of implementation of the strategy was noted.Taking into account the moderating factors of the implementation is necessary for achieving the highest possible degree of implementation fidelity and then, reach the expected beneficial effects.