Dataset Information

Considerations for pooling real-world data as a comparator cohort to a single arm trial: a simulation study on assessment of heterogeneity.

ABSTRACT:

Background

Novel precision medicine therapeutics target increasingly granular, genomically-defined populations. Rare sub-groups make it challenging to study within a clinical trial or single real-world data (RWD) source; therefore, pooling from disparate sources of RWD may be required for feasibility. Heterogeneity assessment for pooled data is particularly complex when contrasting a pooled real-world comparator cohort (rwCC) with a single-arm clinical trial (SAT), because the individual comparisons are not independent as all compare a rwCC to the same SAT. Our objective was to develop a methodological framework for pooling RWD focused on the rwCC use case, and simulate novel approaches of heterogeneity assessment, especially for small datasets.

Methods

We present a framework with the following steps: pre-specification, assessment of dataset eligibility, and outcome analyses (including assessment of outcome heterogeneity). We then simulated heterogeneity assessments for a binary response outcome in a SAT compared to two rwCCs, using standard methods for meta-analysis, and an Adjusted Cochran's Q test, and directly comparing the individual participant data (IPD) from the rwCCs.

Results

We found identical power to detect a true difference for the adjusted Cochran's Q test and the IPD method, with both approaches superior to a standard Cochran's Q test. When assessing the impact of heterogeneity in the null scenario of no difference between the SAT and rwCCs, a lack of statistical power led to Type 1 error inflation. Similarly, in the alternative scenario of a true difference between SAT and rwCCs, we found substantial Type 2 error, with underpowered heterogeneity testing leading to underestimation of the treatment effect.

Conclusions

We developed a methodological framework for pooling RWD sources in the context of designing a rwCC for a SAT. When testing for heterogeneity during this process, the adjusted Cochran's Q test matches the statistical power of IPD heterogeneity testing. Limitations of quantitative heterogeneity testing in protecting against Type 1 or Type 2 error indicate these tests are best used descriptively, and after careful selection of datasets based on clinical/data considerations. We hope these findings will facilitate the rigorous pooling of RWD to unlock insights to benefit oncology patients.

SUBMITTER: Backenroth D

PROVIDER: S-EPMC10464044 | biostudies-literature | 2023 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Considerations for pooling real-world data as a comparator cohort to a single arm trial: a simulation study on assessment of heterogeneity.

Backenroth Daniel D Royce Trevor T Pinheiro Jose J Samant Meghna M Humblet Olivier O

BMC medical research methodology 20230824 1

<h4>Background</h4>Novel precision medicine therapeutics target increasingly granular, genomically-defined populations. Rare sub-groups make it challenging to study within a clinical trial or single real-world data (RWD) source; therefore, pooling from disparate sources of RWD may be required for feasibility. Heterogeneity assessment for pooled data is particularly complex when contrasting a pooled real-world comparator cohort (rwCC) with a single-arm clinical trial (SAT), because the individual ...[more]

PMID: 37620758

Similar Datasets

Project description:BackgroundThe paliperidone palmitate 6-month (PP6M) long-acting injectable formulation is currently the longest dosing interval available for schizophrenia treatment.ObjectiveTo compare treatment outcomes between a real-world external comparator arm (ECA; NeuroBlu database) and the PP6M open-label extension (OLE) clinical trial arm.MethodsThe ECA comprised patients receiving PP 1-month (PP1M) or PP 3-month (PP3M) for ⩾12 months without a relapse. The PP6M OLE arm included patients with PP1M treatment prior to randomization who completed the 12-month double-blind PP6M study on either PP3M or PP6M relapse-free. Inverse probability treatment weighting (IPTW) was used to study time-to-relapse (primary outcome) and change in Clinical Global Impressions-Severity (CGI-S) score (secondary outcome).ResultsAt 24 months, 3.9% (7/178) of patients in the PP6M cohort experienced a relapse versus 15.6% (26/167) in the ECA. Time-to-relapse was longer in the PP6M cohort versus the ECA at 12-, 18-, and 24-months across the different weighting methods; median time-to-relapse was not reached in both cohorts. Hazard ratio (HR) for relapse was significantly lower for the PP6M cohort versus the ECA throughout the duration of the study [HR at 24 months: 0.18 (95% CI: 0.08-0.42), p < 0.001]. At 24 months, change in CGI-S score for the PP6M cohort was 0.76 points lower than the ECA (p < 0.001). Results were similar in a sensitivity analysis using propensity score matching (PSM); IPTW resulted in larger sample sizes in balanced dataset than PSM.ConclusionConsistent findings across weighting and matching methods suggest PP6M efficacy in reducing and delaying relapses and long-term symptom control compared to PP1M/PP3M in usual-care settings. Additional confounds, such as greater illness severity and more frequent comorbidities and comedications in the ECA, were not fully controlled by the applied statistical methods. Future real-world studies directly comparing PP6M with PP3M/PP1M and adjusting for these confounders are warranted.

Project description:BackgroundLarge prospective cohort studies have been fruitful for identifying exposure-disease associations. In a cohort where biospecimens (e.g., blood, urine) were collected at enrollment, analysts can exploit a case-cohort approach: Biospecimens from a random sample of cohort participants, called the "subcohort," plus a sample of incident cases that were not part of the subcohort are assayed. Reusing subcohort data for multiple disease outcomes can reduce costs and conserve specimen archives. Pooling biospecimen samples before assay could both save money and reduce depletion of the archive but has not been studied for cohort studies.ObjectivesWe develop and evaluate a biospecimen pooling strategy for case-cohort analyses that relate an exposure to risk of a rare disease.MethodsOur approach involves constructing pooling sets for cases not in the subcohort after grouping them according to time of diagnosis (e.g., age). In contrast, members of the subcohort are grouped by age at entry before constructing pooling sets. The analyst then fits a logistic regression model that jointly stratifies by age at risk and pooling set size and adjusts for confounders. We used simulations (288 sampling scenarios with 1,000 simulated datasets each) to evaluate the performance of this approach for several sizes of pooling sets and illustrated its application to environmental epidemiologic studies by reanalyzing Sister Study data.ResultsParameter estimates were nearly unbiased, and 95% confidence intervals constructed using a bootstrap estimate of the standard error performed well. In statistical tests also based on the bootstrap standard error, pooling up to 8 specimens per pool caused only modest loss of power. Assigning more cohort members to the subcohort and commensurately increasing the number of specimens per pool improved power and precision substantially while reducing the number of assays.DiscussionWhen using case-cohort analysis to study disease outcomes in relation to exposures assessed using biospecimens in a cohort study, epidemiologists should consider biospecimen pooling as a way to improve statistical power, conserve irreplaceable archives, and save money. https://doi.org/10.1289/EHP14476.

Project description:BackgroundMulti-institution electronic health records (EHR) are a rich source of real world data (RWD) for generating real world evidence (RWE) regarding the utilization, benefits and harms of medical interventions. They provide access to clinical data from large pooled patient populations in addition to laboratory measurements unavailable in insurance claims-based data. However, secondary use of these data for research requires specialized knowledge and careful evaluation of data quality and completeness. We discuss data quality assessments undertaken during the conduct of prep-to-research, focusing on the investigation of treatment safety and effectiveness.MethodsUsing the National COVID Cohort Collaborative (N3C) enclave, we defined a patient population using criteria typical in non-interventional inpatient drug effectiveness studies. We present the challenges encountered when constructing this dataset, beginning with an examination of data quality across data partners. We then discuss the methods and best practices used to operationalize several important study elements: exposure to treatment, baseline health comorbidities, and key outcomes of interest.ResultsWe share our experiences and lessons learned when working with heterogeneous EHR data from over 65 healthcare institutions and 4 common data models. We discuss six key areas of data variability and quality. (1) The specific EHR data elements captured from a site can vary depending on source data model and practice. (2) Data missingness remains a significant issue. (3) Drug exposures can be recorded at different levels and may not contain route of administration or dosage information. (4) Reconstruction of continuous drug exposure intervals may not always be possible. (5) EHR discontinuity is a major concern for capturing history of prior treatment and comorbidities. Lastly, (6) access to EHR data alone limits the potential outcomes which can be used in studies.ConclusionsThe creation of large scale centralized multi-site EHR databases such as N3C enables a wide range of research aimed at better understanding treatments and health impacts of many conditions including COVID-19. As with all observational research, it is important that research teams engage with appropriate domain experts to understand the data in order to define research questions that are both clinically important and feasible to address using these real world data.

Dataset Information

Considerations for pooling real-world data as a comparator cohort to a single arm trial: a simulation study on assessment of heterogeneity.

Background

Methods

Results

Conclusions

Publications

Considerations for pooling real-world data as a comparator cohort to a single arm trial: a simulation study on assessment of heterogeneity.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets