Dataset Information

Design analysis indicates Potential overestimation of treatment effects in randomized controlled trials supporting Food and Drug Administration cancer drug approvals.

ABSTRACT:

Objective

Statistical significance drives interpretation of randomized controlled trials (RCTs). We examined the type S error risk-claiming a new drug is falsely beneficial-and exaggeration ratio-how estimated effects differ from true effects-to re-emphasize direction and magnitude of treatment effects.

Study design and setting

We systematically reviewed RCTs supporting Food and Drug Administration (FDA) approval of cancer drugs between 2007 and 2016. We extracted data for overall survival (OS), progression-free survival (PFS), and response outcomes from FDA reviews. We estimated type S error risks and exaggeration ratios by considering replicated RCTs of equal size and a range of true effects.

Results

We analyzed 42 RCTs for 39 approved drugs. Across 38 RCTs reporting OS, the median type S error risk was 0.00% (Q1-Q3, 0.00-0.01%) and 3.56% (0.40-6.74%), for true hazard ratios of 0.7 and 0.9, respectively, indicating confidence in effect direction. The corresponding exaggeration ratios were 1.09 (1.01-1.11) and 1.30 (1.13-1.42), indicating median overestimations of 9% and 30%. Similar results held for PFS and response outcomes.

Conclusions

The type S error risk and exaggeration ratio provide additional insights into the replicability of RCTs. Our analyses also quantify the winner's curse, in which pivotal RCTs tend toward overoptimism.

SUBMITTER: Lord EM

PROVIDER: S-EPMC8978539 | biostudies-literature | 2018 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Design analysis indicates Potential overestimation of treatment effects in randomized controlled trials supporting Food and Drug Administration cancer drug approvals.

Lord Emily M EM Weir Isabelle R IR Trinquart Ludovic L

Journal of clinical epidemiology 20180702

<h4>Objective</h4>Statistical significance drives interpretation of randomized controlled trials (RCTs). We examined the type S error risk-claiming a new drug is falsely beneficial-and exaggeration ratio-how estimated effects differ from true effects-to re-emphasize direction and magnitude of treatment effects.<h4>Study design and setting</h4>We systematically reviewed RCTs supporting Food and Drug Administration (FDA) approval of cancer drugs between 2007 and 2016. We extracted data for overall ...[more]

PMID: 30297036

Similar Datasets

Project description:ImportanceWhile there have been multiple assessments of clinical trials leading to anticancer drug approvals by the US Food and Drug Administration (FDA), the cumulative percentage of approvals based on trials with a limitation remains uncertain.ObjectiveTo assess the percentage of clinical trials with limitations in 4 domains-lack of randomization, lack of significant overall survival advantage, inappropriate use of crossover, and use of suboptimal control arms-that led to FDA approvals from June 30, 2014, to July 31, 2019.Design, setting, and participantsThis observational analysis included all anticancer drug indications approved by the FDA from June 30, 2014, through July 31, 2019. All indications were investigated, and each clinical trial was evaluated for design, enrollment period, primary end points, and presence of a limitation in the domains of interest. The standard-of-care therapy was determined by evaluating the literature and published guidelines 1 year prior to the start of clinical trial enrollment. Crossover was examined and evaluated for optimal use. The percentage of approvals based on clinical trials with any or all limitations of interest was then calculated.Main outcomes and measuresEstimated percentage of clinical trials with limitations of interest that led to an anticancer drug marketing authorization by the FDA.ResultsA total of 187 trials leading to 176 approvals for 75 distinct novel anticancer drugs by the FDA were evaluated. Sixty-four (34%) were single-arm clinical trials, and 123 (63%) were randomized clinical trials. A total of 125 (67%) had at least 1 limitation in the domains of interest; 60 of the 125 trials (48%) were randomized clinical trials. Of all 123 randomized clinical trials, 37 (30%) lacked overall survival benefit, 31 (25%) had a suboptimal control, and 17 (14%) used crossover inappropriately.Conclusions and relevanceTwo-thirds of cancer drugs are approved based on clinical trials with limitations in at least 1 of 4 essential domains. Efforts to minimize these limitations at the time of clinical trial design are essential to ensure that new anticancer drugs truly improve patient outcomes over current standards.

Project description:ImportanceThe growth of cancer drug spending in the US has outpaced spending in nearly all other sectors, and an increasing proportion of the drug development pipeline is devoted to oncology. In 2018, there was a record number of drugs entering the US market.ObjectiveTo estimate the number of patients with cancer who are eligible for the newly approved drug-indication pairs, and project potential spending and use of the approvals in the US.Design, setting, participantsThis is a retrospective review of 2018 US Food and Drug Administration (FDA) oncology drug approvals with estimation of the eligible population. The cost of new therapy was estimated, and savings from displaced therapies were subtracted. Two-way sensitivity analysis explored uncertainty in pricing and market diffusion. Data were collected between March 1, 2019, and September 30, 2019.ExposuresData related to the cancer drug approval (ie, indications, approval pathway, basis for approval), cancer incidence, and drug price were extracted from publicly available sources, including the FDA, National Cancer Institute, and American Cancer Society websites, as well as the RED BOOK database.Main outcomes and measuresThe primary outcome was the projected net expenditure in the US associated with the new therapies. The secondary outcome described how variable market diffusion and pricing permit expected levels of spending.ResultsA total of 46 oncology approvals were included in the analysis, with 17 novel drugs and 29 new indications. The average price per patient per treatment course was $150 384. From a national perspective and with 100% market diffusion, the projected net expenditure for newly approved drugs was $39.5 billion per year. To maintain the recent trend of cancer drug spending, the 2018 cancer drug approvals need to be used in fewer than 20% of eligible patients.Conclusions and relevanceNew cancer drugs approved by the FDA in 2018 would drastically increase cancer drug spending in the US if used widely. Alternatively, only low-level use of the new drugs is consistent with market forecasting.

Project description:ImportanceDiverse, representative enrollment in pivotal clinical trials is vital to sufficiently power subgroup analyses and ensure equity and validity of trial results.ObjectiveTo evaluate the racial/ethnic representation, trends, and disparities in clinical trials leading to US Food and Drug Administration (FDA) ophthalmology drug approvals from 2000 to 2020.Design, setting, and participantsThis cohort study used data from participants in clinical trials of drugs for neovascular age-related macular degeneration (AMD), open-angle glaucoma (OAG), and expanded indications for diabetic retinopathy (DR) from January 1, 2000, to December 31, 2020. Trial data were sourced from FDA reviews, ClinicalTrials.gov, and relevant linked studies. National expected racial/ethnic proportions were sourced from public National Eye Institute prevalence data as well as published rates scaled using US Census Bureau data.Main outcomes and measuresThe primary outcome measures were the distribution of and change over time in the racial/ethnic proportion of participants in clinical trials leading to FDA approval of drugs for AMD, OAG, and DR.ResultsDuring the 20-year period, 31 clinical trials were identified for 13 medications with 18 410 participants. The distribution of trial participants was different from the expected trial distribution for most approvals with regard to race/ethnicity (12 drugs) and sex (10 drugs). Compared with the first decade (2000-2010), trials conducted in the second decade (2011-2020) showed increases in enrollment of Asian (odds ratio [OR], 2.30; 95% CI, 1.97-2.68; P < .001) and Hispanic or Latinx participants (OR, 1.74; 95% CI, 1.49-2.03; P < .001) for AMD, Asian participants (OR, 2.21; 95% CI, 1.46-3.42; P < .001) for DR, and Black (OR, 1.60; 95% CI, 1.43-1.78; P < .001) and Hispanic or Latinx participants (OR, 10.31; 95% CI, 8.05-13.35; P < .001) for OAG. There was a decrease in Black participants in DR trials (OR, 0.58; 95% CI, 0.42-0.79; P < .001). Based on these trends, the enrollment incidence ratio is expected to worsen by 2050, with overrepresentation of white participants vs underrepresentation of Black and Hispanic or Latinx participants in trials of drugs for AMD (1.08 vs 0.04 vs 0.77), DR (1.83 vs 0.87 vs 0.59), and OAG (1.62 vs 0.90 vs 0.37).Conclusions and relevanceIn this cohort study, Black, Hispanic or Latinx, and other non-White participants were underrepresented in clinical trials leading to FDA ophthalmology drug approvals compared with the expected disease burden and racial/ethnic distribution in the US. Although there was meaningful improvement from 2000 to 2020, further efforts to increase minority enrollment in clinical trials seem to be warranted.

Project description:ImportanceThe size of estimated treatment effects on the basis of which the US Food and Drug Administration (FDA) has approved drugs and devices with data from nonrandomized clinical trials (non-RCTs) remains unknown.ObjectivesTo determine how often the FDA has authorized novel interventions based on non-RCTs and to assess whether there is an association of the magnitude of treatment effects with FDA requirements for additional testing in randomized clinical trials (RCTs).Data sourcesOverall, 606 drug applications for the Breakthrough Therapy designation from its inception in January 2012 were downloaded from the FDA website in January 2017 and August 2018, and 71 medical device applications for the Humanitarian Device Exemption from its inception in June 1996 were downloaded in August 2017.Study selectionApproved applications based on non-RCTs were included; RCTs, studies with insufficient information, duplicates, and safety data were excluded.Data extraction and synthesisData were extracted by 2 independent investigators. A statistical association of the magnitude of estimated effect (expressed as an odds ratio) with FDA requests for RCTs was assessed. The data were also meta-analyzed to evaluate the differences in odds ratios between applications that required further testing and those that did not. The results are reported according to Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines.Main outcomes and measuresDisease, laboratory, and patient-related outcomes, including disease response or patient survival, were considered.ResultsAmong 677 drug and medical device applications, 68 (10.0%) were approved by the FDA based on non-RCTs. Estimates of effects were larger when no further RCTs were required (mean natural logarithm of the odds ratios, 2.18 vs 1.12; odds ratios, 8.85 vs 3.06; P = .03). The meta-analysis results confirmed these findings: estimated effects were approximately 2.5-fold higher for treatments or devices that were approved based on non-RCTs than for treatments or devices for which further testing in RCTs was required (6.30 [95% CI, 4.38-9.06] vs 2.46 [95% CI, 1.70-3.56]; P < .001). Overall, 9 of 677 total applications (1.3%) that were approved on the basis of non-RCTs had relative risks of 10 or greater and 12 (1.7%) had relative risks of 5 or greater. No clear threshold above which the FDA approved interventions based on the magnitude of estimated effect alone was detected.Conclusions and relevanceIn this study, estimated magnitudes of effect were larger among studies for which the FDA did not require RCTs compared with studies for which it did. There was no clear threshold of treatment effect above which no RCTs were requested.

Project description:ImportanceSince the introduction of the Fast Track designation in 1988, the number of special regulatory programs available for the approval of new drugs and biologics by the US Food and Drug Administration (FDA) has increased, offering the agency flexibility with respect to evidentiary requirements.ObjectiveTo characterize pivotal efficacy trials supporting the approval of new drugs and biologics during the past 3 decades.Design, setting, and participantsThis cross-sectional study included 273 new drugs and biologics approved by the FDA for 339 indications from 1995 to 1997, from 2005 to 2007, and from 2015 to 2017.Main outcomes and measuresTherapeutics were classified by product type and therapeutic area as well as orphan designation and use of special regulatory programs, such as Priority Review and Accelerated Approval. Pivotal trials were characterized by use of randomization, blinding, types of comparators, primary end points, number of treated patients, and trial duration, both individually and aggregated by each indication approval.ResultsA total of 273 new drugs and biologics were approved by the FDA in these 3 periods (107 [39.2%] in 1995-1997; 57 [20.9%] in 2005-2007; and 109 [39.9%] in 2015-2017), representing 339 indications (157 [46.3%], 64 [18.9%], and 118 [34.8%], respectively). The proportion of therapeutic approvals using at least 1 special regulatory program increased (37 [34.6%] in 1995-1997; 33 [57.9%] in 2005-2007; and 70 [64.2%] in 2015-2017), as did indication approvals receiving an orphan designation (20 [12.7%] in 1995-1997; 17 [26.6%] in 2005-2007, and 45 [38.1%] in 2015-2017). The most common therapeutic areas differed over time (infectious disease, 53 [33.8%] in 1995-1997 vs cancer, 32 [27.1%] in 2015-2017). When considering the aggregate pivotal trials supporting each indication approval, the proportion of indications supported by at least 2 pivotal trials decreased (80.6% [95% CI, 72.6%-87.2%] in 1995-1997; 60.3% [95% CI, 47.2%-72.4%] in 2005-2007; and 52.8% [95% CI, 42.9%-62.6%] in 2015-2017; P < .001). The proportion of indications supported by only single-group pivotal trials increased (4.0% [95% CI, 1.3%-9.2%] in 1995-1997; 12.7% [95% CI, 5.6%-23.5%] in 2005-2007; and 17.0% [95% CI, 10.4%-25.5%] in 2015-2017; P = .001), whereas the proportion supported by at least 1 pivotal trial of 6 months' duration increased (25.8% [95% CI, 18.4%-34.4%] in 1995-1997; 34.9% [95% CI, 23.3%-48.0%] in 2005-2007; and 46.2% [95% CI, 36.5%-56.2%] in 2015-2017; P = .001).Conclusions and relevanceIn this study, more recent FDA approvals of new drugs and biologics were based on fewer pivotal trials, which, when aggregated by indication, had less rigorous designs but longer trial durations, suggesting an ongoing need for continued evaluation of therapeutic safety and efficacy after approval.

Dataset Information

Design analysis indicates Potential overestimation of treatment effects in randomized controlled trials supporting Food and Drug Administration cancer drug approvals.

Objective

Study design and setting

Results

Conclusions

Publications

Design analysis indicates Potential overestimation of treatment effects in randomized controlled trials supporting Food and Drug Administration cancer drug approvals.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets