Applying the PRECIS criteria to describe three effectiveness trials of weight loss in obese patients with comorbid conditions.
ABSTRACT: To characterize Practice-Based Opportunities for Weight Reduction (POWER) trials along the pragmatic-explanatory continuum.The POWER trials consist of three individual studies that target obesity treatment in primary care settings.Using the PRagmatic Explanatory Continuum Indicator Summary (PRECIS) criteria, nine reviewers independently scored each trial.Average and median ratings, inter-rater reliability, and relationships to additional ratings of the extent to which study designs were explanatory (i.e., efficacy) versus pragmatic (i.e., practical) and related to external validity were determined.One trial was consistently rated as being significantly more pragmatic than the others (R(2) =0.43, p< .001), although all three were in the moderate range on the PRECIS scales. Ratings varied across PRECIS dimensions, being most pragmatic on comparison condition and primary outcome. Raters, although undergoing training and using identical definitions, scored their own study as more pragmatic than the other studies/interventions.These results highlight the need for more comprehensive reporting on PRECIS and related criteria for research translation. The PRECIS criteria provide a richer understanding of the POWER studies. It is not clear whether the original criteria are sufficient to provide a comprehensive profile.
Project description:This study tests the feasibility of applying the pragmatic-explanatory continuum indicator summary (version "PRECIS-2") tool to randomized controlled trials of Chinese herbal medicine. A search was conducted to identify potentially eligible randomized controlled trials. Using the PRECIS-2 tool, assessment of trials was performed independently by 2 evaluators using a scale of 1-5 for each criterion (1 = maximal efficacy, 5 = maximal effectiveness). A total of 7,166 reports were retrieved from databases and 159 were included in the full text. Though PRECIS-2 describes quantitative scoring in detail, evaluators were uncertain about several specific operationalizations and found high evaluator variation in the first independent ratings. After discussion and reaching consensus, inter-evaluator reliability improved. For PRECIS-2 ratings over time, there was no evidence that the design and performance of RCTs of CHM paid more attention to "efficacy" criteria after the implementation of PRECIS (all P > 0.05). More research is needed to establish the easiest and most useful tool to distinguish between effectiveness and efficacy results.
Project description:Numerous explanatory randomized trials support the efficacy of chronic disease interventions, including smoking cessation treatments. However, there is often inadequate adoption of these interventions for various reasons, one being the limitation of generalizability of the explanatory studies in real-world settings. Randomized controlled trials can be rated as more explanatory versus pragmatic along 10 dimensions. Pragmatic randomized clinical trials generate more realistic estimates of effectiveness with greater relevance to clinical practice and for health resource allocation decisions. However, there is no clear method to scale each dimension during the trial design phase to ensure that the design matches the intended purpose of the study.We designed a pragmatic, randomized, controlled study to maximize external validity by addressing several barriers to smoking cessation therapy in ambulatory care. We analyzed our design and methods using the recently published 'Pragmatic-Explanatory Continuum Indicatory Summary (PRECIS)' tool, a qualitative method to assess trial design across 10 domains. We added a 20-point numerical rating scale and a modified Delphi process to improve consensus in rating these domains.After two rounds of review, there was consensus on all 10 domains of study design. No single domain was scored as either fully pragmatic or fully explanatory; but overall, the study scored high on pragmatism.This addition to the PRECIS tool may assist other trial designers working with interdisciplinary co-investigators to rate their study design while building consensus.
Project description:BACKGROUND:If you want to know which of two or more healthcare interventions is most effective, the randomised controlled trial is the design of choice. Randomisation, however, does not itself promote the applicability of the results to situations other than the one in which the trial was done. A tool published in 2009, PRECIS (PRagmatic Explanatory Continuum Indicator Summaries) aimed to help trialists design trials that produced results matched to the aim of the trial, be that supporting clinical decision-making, or increasing knowledge of how an intervention works. Though generally positive, groups evaluating the tool have also found weaknesses, mainly that its inter-rater reliability is not clear, that it needs a scoring system and that some new domains might be needed. The aim of the study is to: Produce an improved and validated version of the PRECIS tool. Use this tool to compare the internal validity of, and effect estimates from, a set of explanatory and pragmatic trials matched by intervention. METHODS:The study has four phases. Phase 1 involves brainstorming and a two-round Delphi survey of authors who cited PRECIS. In Phase 2, the Delphi results will then be discussed and alternative versions of PRECIS-2 developed and user-tested by experienced trialists. Phase 3 will evaluate the validity and reliability of the most promising PRECIS-2 candidate using a sample of 15 to 20 trials rated by 15 international trialists. We will assess inter-rater reliability, and raters' subjective global ratings of pragmatism compared to PRECIS-2 to assess convergent and face validity. Phase 4, to determine if pragmatic trials sacrifice internal validity in order to achieve applicability, will compare the internal validity and effect estimates of matched explanatory and pragmatic trials of the same intervention, condition and participants. Effect sizes for the trials will then be compared in a meta-regression. The Cochrane Risk of Bias scores will be compared with the PRECIS-2 scores of pragmatism. DISCUSSION:We have concrete suggestions for improving PRECIS and a growing list of enthusiastic individuals interested in contributing to this work. By early 2014 we expect to have a validated PRECIS-2.
Project description:The National Institutes of Health (NIH) Health Care Systems Research Collaboratory (NIH Collaboratory) seeks to produce generalizable knowledge about the conduct of pragmatic research in health systems. This analysis applied the PRECIS-2 pragmatic trial criteria to five NIH Collaboratory pragmatic trials to better understand 1) the pragmatic aspects of the design and implementation of treatments delivered in real world settings and 2) the usability of the PRECIS-2 criteria for assessing pragmatic features across studies and across time.Using the PRECIS-2 criteria, five pragmatic trials were each rated by eight raters. For each trial, we reviewed the original grant application and a required progress report written at the end of a 1-year planning period that included changes to the protocol or implementation approach. We calculated median scores and interrater reliability for each PRECIS domain and for the overall trial at both time points, as well as the differences in scores between the two time points. We also reviewed the rater comments associated with the scores.All five trials were rated to be more pragmatic than explanatory, with comments indicating that raters generally perceived them to closely mirror routine clinical care across multiple domains. The PRECIS-2 domains for which the trials were, on average, rated as most pragmatic on the 1 to 5 scale at the conclusion of the planning period included primary analysis (mean?=?4.7 (range?=?4.5 to 4.9)), recruitment (4.3 (3.6 to 4.8)), eligibility (4.1 (3.4 to 4.8)), setting (4.1 (4.0 to 4.4)), follow-up (4.1 (3.4 to 4.9)), and primary outcome (4.1 (3.5 to 4.9)). On average, the less pragmatic domains were organization (3.3 (2.6 to 4.4)), flexibility of intervention delivery (3.5 (2.1-4.5)), and flexibility of intervention adherence (3.8 (2.8-4.5)). Interrater agreement was modest but statistically significant for four trials (Gwet's AC1 statistic range 0.23 to 0.40) and the intraclass correlation coefficient ranged from 0.05 to 0.31. Rating challenges included assigning a single score for domains that may relate to both patients and care settings (that is, eligibility or recruitment) and determining to what extent aspects of complex research interventions differ from usual care.These five trials in diverse healthcare settings were rated as highly pragmatic using the PRECIS-2 criteria. Applying the tool generated insightful discussion about real-world design decisions but also highlighted challenges using the tool. PRECIS-2 raters would benefit from additional guidance about how to rate the interwoven patient and practice-level considerations that arise in pragmatic trials.Clinicaltrials.gov trial registrations: NCT02019225 , NCT01742065 , NCT02015455 , NCT02113592 , NCT02063867 .
Project description:BACKGROUND:PRECIS-2 (PRagmatic Explanatory Continuum Indicator Summary-2) can assess how clinical trial design decisions (along the explanatory-pragmatic continuum) influence the applicability of trial results to intended stakeholders. The tool has been used to assess features of trials during the trial design phase and also upon completion. The ongoing PRagmatic trial Of Video Education in Nursing homes (PROVEN), which is evaluating the effectiveness of a suite of videos to improve advance care planning, is one of the first large pragmatic, cluster randomized trials within nursing home health care systems. While certain features of pragmatic trials remain static once designed (e.g., recruitment, outcomes), successful implementation of a system-wide program requires on-going evaluation and adaptation. This report's objectives were to apply PRECIS-2 in a novel manner during the actual conduct of the PROVEN trial to assess how dynamic adaptations shifted implementation to either a more explanatory or a more pragmatic approach. METHODS:We assessed PROVEN's protocol as initially designed according to the three PRECIS-2 domains pertinent to implementation: (1) Organization, (2) Flexibility-Delivery, and (3) Flexibility-Adherence. We then applied this framework to conduct a formative evaluation of decisions made while the trial was ongoing to adapt the implementation approach along the pragmatic versus the explanatory continuum in response to emergent challenges. RESULTS:Based on the PRECIS-2 rubric, the initial design of the PROVEN implementation approach reflected a hybrid of pragmatic and explanatory features. Most notably, within the Flexibility-Delivery, the trial had a relatively pragmatic approach to protocol delivery by front-line nursing home providers, balanced with a more explanatory approach to protocol monitoring enabled by the analytic capabilities of the research team. This more intensive monitoring proved critical in revealing implementation problems once the study began. Dynamic adaptations made in response to these challenges generally reflected shifts to more explanatory approaches within the Flexibility-Delivery and Flexibility-Adherence domains including ever more intensive compliance monitoring, as well as detailed coaching of front-line providers delivering the intervention by the research team. CONCLUSIONS:Pragmatic trials conducted in the nursing home setting may benefit from a more dynamic approach to implementation. Allowing fluidity between pragmatic and explanatory features may still preserve the trial's applicability to intended stakeholders' needs. PRECIS-2 provides a useful formative evaluation tool to assess these adaptations in "real-time." TRIAL REGISTRATION:US National Library of Medicine, ClinicalTrials.gov, ID: NCT02612688 . Registered on 19 November 2015.
Project description:<h4>Background</h4>First articulated by Schwartz and Lellouch (1967), randomized controlled trials (RCTs) can be conceptualized along a continuum from more explanatory to more pragmatic. The purpose and intent of the former is to test interventions under ideal contexts, and the purpose and intent of the latter is to test interventions in real-world contexts. The PRagmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2) is a validated tool that helps researchers make decisions about the elements of the trial to match the overall purpose and intent of the trial along the continuum. The PRECIS-2 tool has guided the design of hundreds of RCTs. However, a few aspects of the tool would benefit from greater clarity, including its application to provider-focused implementation trials rather than patient-focused intervention trials.<h4>Main text</h4>We describe the newly developed PRECIS-2-Provider Strategies (PRECIS-2-PS) tool, an extension of the PRECIS-2 tool, which has been adapted for trials testing provider-focused strategies. We elaborate on nine domains that can make a provider-focused trial more explanatory or more pragmatic, including eligibility, recruitment, setting, implementation resources, flexibility of provider strategies, flexibility of intervention, data collection, primary outcome, and primary analysis. We detail the complementary roles that researchers and stakeholders play in the trial design phase, with implications for generalizability of trial results to the contexts in which they are intended to be applied.<h4>Conclusions</h4>The PRECIS-2-PS tool is designed to help research and practice teams plan for provider-focused trials that reflect the overall intent and purpose of the trial. The tool has potential to help advance the science of provider-focused strategies across a range of trials, with the ultimate goal of facilitating the adoption, integration, and sustainability of provider-focused strategies outside the context of trials.
Project description:BACKGROUND:Recommendations for good clinical practice have been reported to be difficult to apply in real life by primary care clinicians. This could be because the clinical trials at the origin of the guidelines are based on explanatory trials, conducted under ideal conditions not reflecting the reality of primary care, rather than pragmatic trials conducted under real-life conditions. The objective of this study was to evaluate how pragmatic are the clinical trials used to build the French High Authority of Health's recommendations on the management of type II diabetes. METHODS:Trials from the 2013 Cochrane meta-analysis that led to the 2013 French High Authority of Health's recommendations on the management of type II diabetes were selected. Each trial was analysed by applying the PRECIS-2 tool to evaluate whether the trial was pragmatic or explanatory, according to the nine domains of PRECIS-2. Each domain was scored between 1 (very explanatory) and 5 (very pragmatic) by two blinded researchers, and consensus was reached with a third researcher in case of discrepancy. Median scores were calculated for each of the nine domains. RESULTS:Twenty-three articles were analysed. Eight out of nine domains - namely eligibility, recruitment, setting, organisation, flexibility of delivery, flexibility of adherence, follow-up, and primary outcome - had a median score of less than 3, indicating a more explanatory design. Only the primary analysis domain had a score indicating a more pragmatic approach (median score of 4). In more than 25% of the articles, data to score the domains of recruitment, flexibility of delivery, flexibility of adherence, and primary analysis were missing. CONCLUSIONS:Trials used to build French recommendations for good clinical practice for the management of type 2 diabetes in primary care were more explanatory than pragmatic. Policy-makers should encourage the funding of pragmatic trials to evaluate the different strategies proposed for managing the patient's treatment according to HbA1C levels and give clinicians feasible recommendations.
Project description:It is important to consider the degree to which studies are explanatory versus pragmatic to understand the implications of their findings for patients, healthcare professionals, and policymakers. Pragmatic trials test the effectiveness of interventions in real-world conditions; explanatory trials test for efficacy under ideal conditions. The Consortium of Hospitals Advancing Research on Tobacco (CHART) is a network of seven NIH-funded trials designed to identify effective programs that can be widely implemented in routine clinical practice.A cross-sectional analysis of CHART trial study designs was conducted to place each study on the pragmatic-explanatory continuum. After reliability training, six raters independently scored each CHART study according to ten PRagmatic Explanatory Continuum Indicator Summary (PRECIS) dimensions, which covered participant eligibility criteria, intervention flexibility, practitioner expertise, follow-up procedures, participant compliance, practitioner adherence, and outcome analyses. Means and SDs were calculated for each dimension of each study, with lower scores representing more pragmatic elements. Results were plotted on "spoke and wheel" diagrams. The rating process and analyses were performed in October 2014 to September 2015.All seven CHART trials tended toward the pragmatic end of the spectrum, although there was a range from 0.76 (SD=0.23) to 1.85 (SD=0.58). Most studies included some explanatory design elements.CHART findings should be relatively applicable to clinical practice. Funders and reviewers could integrate PRECIS criteria into their guidelines to better facilitate pragmatic research. CHART study protocols, coupled with scores reported here, may help readers improve the design of their own pragmatic trials.
Project description:INTRODUCTION:There has been increasing interest in pragmatic trials methodology. As a result, tools such as the Pragmatic-Explanatory Continuum Indicator Summary-2 (PRECIS-2) are being used prospectively to help researchers design randomised controlled trials (RCTs) within the pragmatic-explanatory continuum. There may be value in applying the PRECIS-2 tool retrospectively in a systematic review setting as it could provide important information about how to pool data based on the degree of pragmatism. OBJECTIVES:To investigate the role of pragmatism as a source of heterogeneity in systematic reviews by (1) identifying systematic reviews with meta-analyses of RCTs that have moderate to high heterogeneity, (2) applying PRECIS-2 to RCTs of systematic reviews, (3) evaluating the inter-rater reliability of PRECIS-2, (4) determining how much of this heterogeneity may be explained by pragmatism. METHODS:A cross-sectional methodological review will be conducted on systematic reviews of RCTs published in the Cochrane Library from 1 January 2014 to 1 January 2017. Included systematic reviews will have a minimum of 10 RCTs in the meta-analysis of the primary outcome and moderate to substantial heterogeneity (I2?50%). Of the eligible systematic reviews, a random selection of 10 will be included for quantitative evaluation. In each systematic review, RCTs will be scored using the PRECIS-2 tool, in duplicate. Agreement between raters will be measured using the intraclass correlation coefficient. Subgroup analyses and meta-regression will be used to evaluate how much variability in the primary outcome may be due to pragmatism. DISSEMINATION:This review will be among the first to evaluate the PRECIS-2 tool in a systematic review setting. Results from this research will provide inter-rater reliability information about PRECIS-2 and may be used to provide methodological guidance when dealing with pragmatism in systematic reviews and subgroup considerations. On completion, this review will be submitted to a peer-reviewed journal for publication.
Project description:BACKGROUND:The Endovascular Acute Stroke Intervention (EASI) trial was conceived as a pragmatic care trial, designed to integrate trial methods with clinical practice. Reporting the EASI experience was met with objections and criticisms during peer review concerning both scientific and ethical issues. Our goal is to discuss these criticisms in order to promote the pragmatic approach of care trials in outcome-based medical care. METHODS:The comments and criticisms of 11 reviewers from 5 journals were collected and analyzed. The EASI protocol was also compared to the protocols of seven thrombectomy trials using the pragmatic-explanatory continuum indicator summary (PRECIS). RESULTS:Main criticisms of EASI concerned selection criteria that were judged to be too vague and too inclusive, brain and vascular imaging methods that were not sufficiently prescribed by protocol, lack of blinding of outcome assessment, and lack of power. EASI was at the pragmatic end of the spectrum of thrombectomy trials. CONCLUSION:The pragmatic care trial methodology is not currently well-established. More work needs to be done to integrate scientific methods and ethical care in the best medical interest of current patients.