Dataset Information

Comparing Two-way Fixed Effects and New Estimators for Difference-in-Differences: A Simulation Study and Empirical Example.

ABSTRACT:

Background

Two-way fixed effects methods have been used to estimate effects of policies adopted in different places over time, but they can provide misleading results when effects are heterogeneous or dynamic, and alternate methods have been proposed.

Methods

We compared methods for estimating the average treatment effect on the treated (ATT) under staggered adoption of policies, including two-way fixed effects, group-time ATT, cohort ATT, and target-trial approaches. We applied each method to assess the impact of Medicaid expansion on preterm birth using the National Center for Health Statistics' birth records. We compared each estimator's performance in a simulation parameterized to mimic the empirical example. We generated constant, heterogeneous, and dynamic effects and calculated bias, mean squared error, and confidence interval coverage of each estimator across 1000 iterations.

Results

Two-way fixed effects estimated that Medicaid expansion increased the risk of preterm birth (risk difference [RD], 0.12; 95% CI = 0.02, 0.22), while the group-time ATT, cohort ATT, and target-trial approaches estimated protective or null effects (group-time RD, -0.16; 95% CI = -0.58, 0.26; cohort RD, -0.02; 95% CI = -0.46, 0.41; target trial RD, -0.16; 95% CI = -0.59, 0.26). In simulations, two-way fixed effects performed well when treatment effects were constant and less well under heterogeneous and dynamic effects.

Conclusions

We demonstrated why new approaches perform better than two-way fixed effects when treatment effects are heterogeneous or dynamic under a staggered policy adoption design, and created simulation and analysis code to promote understanding and wider use of these methods in the epidemiologic literature.

SUBMITTER: Goin DE

PROVIDER: S-EPMC10771800 | biostudies-literature | 2023 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparing Two-way Fixed Effects and New Estimators for Difference-in-Differences: A Simulation Study and Empirical Example.

Goin Dana E DE Riddell Corinne A CA

Epidemiology (Cambridge, Mass.) 20230316 4

<h4>Background</h4>Two-way fixed effects methods have been used to estimate effects of policies adopted in different places over time, but they can provide misleading results when effects are heterogeneous or dynamic, and alternate methods have been proposed.<h4>Methods</h4>We compared methods for estimating the average treatment effect on the treated (ATT) under staggered adoption of policies, including two-way fixed effects, group-time ATT, cohort ATT, and target-trial approaches. We applied e ...[more]

PMID: 36943806

Similar Datasets

Project description:BackgroundReliable evaluations of state-level policies are essential for identifying effective policies and informing policymakers' decisions. State-level policy evaluations commonly use a difference-in-differences (DID) study design; yet within this framework, statistical model specification varies notably across studies. More guidance is needed about which set of statistical models perform best when estimating how state-level policies affect outcomes.MethodsMotivated by applied state-level opioid policy evaluations, we implemented an extensive simulation study to compare the statistical performance of multiple variations of the two-way fixed effect models traditionally used for DID under a range of simulation conditions. We also explored the performance of autoregressive (AR) and GEE models. We simulated policy effects on annual state-level opioid mortality rates and assessed statistical performance using various metrics, including directional bias, magnitude bias, and root mean squared error. We also reported Type I error rates and the rate of correctly rejecting the null hypothesis (e.g., power), given the prevalence of frequentist null hypothesis significance testing in the applied literature.ResultsMost linear models resulted in minimal bias. However, non-linear models and population-weighted versions of classic linear two-way fixed effect and linear GEE models yielded considerable bias (60 to 160%). Further, root mean square error was minimized by linear AR models when we examined crude mortality rates and by negative binomial models when we examined raw death counts. In the context of frequentist hypothesis testing, many models yielded high Type I error rates and very low rates of correctly rejecting the null hypothesis (< 10%), raising concerns of spurious conclusions about policy effectiveness in the opioid literature. When considering performance across models, the linear AR models were optimal in terms of directional bias, root mean squared error, Type I error, and correct rejection rates.ConclusionsThe findings highlight notable limitations of commonly used statistical models for DID designs, which are widely used in opioid policy studies and in state policy evaluations more broadly. In contrast, the optimal model we identified--the AR model--is rarely used in state policy evaluation. We urge applied researchers to move beyond the classic DID paradigm and adopt use of AR models.

Project description:BackgroundSimulation-based training allows trainees to experiment during training and end-of-training tests could increase motivation and retention. The aim of this trial was to determine if a simulation-based training program including directed self-regulated learning and post-testing improved clinical outcomes compared to a traditional simulation-based training program.MethodsA randomized trial was conducted involving 32 participants without prior experience in endoscopic procedures. The intervention group practiced independently in a simulation centre and got a post-test whereas the control group received traditional instructions and demonstrations before being allowed to practice. Three weeks after the intervention the participants performed cystoscopies on two consecutive patients. Clinical performance was assessed using a global rating scale (GRS) with established evidence of validity. Independent samples t-test, Cronbach's α, Pearson's r, and paired samples t-test were used for statistical analysis.ResultsTwenty-five participants performed two cystoscopies on patients. There was no significant difference between the two study groups with respect to mean GRS of performance (p = 0.63, 95 % CI; -2.4-3.9). The internal consistency of the global rating scale was high, Cronbach's α = 0.91. Participants from both study groups demonstrated significant improvement between the first and second clinical procedures (p = 0.004, 95 % CI, 0.8-3.5). Eight (32%) and 15 (60%) participants demonstrated adequate clinical skills in their first and second procedure, respectively.ConclusionsNo significant differences were found on the clinical transfer when comparing the two programs. Neither of our training programs was able to ensure consistent, competent performance on patients and this finding could serve as an important argument for simulation-based mastery learning where all training continues until a pre-defined level of proficiency is met.Trial registrationsThe trial was submitted before enrolment of participants to the Regional Scientific Ethics Committee of the Capital Region which established that ethical approval was not necessary (H-4-2014-122). The trial was registered at Clinicaltrials.gov (NCT02411747).

Dataset Information

Comparing Two-way Fixed Effects and New Estimators for Difference-in-Differences: A Simulation Study and Empirical Example.

Background

Methods

Results

Conclusions

Publications

Comparing Two-way Fixed Effects and New Estimators for Difference-in-Differences: A Simulation Study and Empirical Example.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets