Bias in Research Grant Evaluation Has Dire Consequences for Small Universities.
ABSTRACT: Federal funding for basic scientific research is the cornerstone of societal progress, economy, health and well-being. There is a direct relationship between financial investment in science and a nation's scientific discoveries, making it a priority for governments to distribute public funding appropriately in support of the best science. However, research grant proposal success rate and funding level can be skewed toward certain groups of applicants, and such skew may be driven by systemic bias arising during grant proposal evaluation and scoring. Policies to best redress this problem are not well established. Here, we show that funding success and grant amounts for applications to Canada's Natural Sciences and Engineering Research Council (NSERC) Discovery Grant program (2011-2014) are consistently lower for applicants from small institutions. This pattern persists across applicant experience levels, is consistent among three criteria used to score grant proposals, and therefore is interpreted as representing systemic bias targeting applicants from small institutions. When current funding success rates are projected forward, forecasts reveal that future science funding at small schools in Canada will decline precipitously in the next decade, if skews are left uncorrected. We show that a recently-adopted pilot program to bolster success by lowering standards for select applicants from small institutions will not erase funding skew, nor will several other post-evaluation corrective measures. Rather, to support objective and robust review of grant applications, it is necessary for research councils to address evaluation skew directly, by adopting procedures such as blind review of research proposals and bibliometric assessment of performance. Such measures will be important in restoring confidence in the objectivity and fairness of science funding decisions. Likewise, small institutions can improve their research success by more strongly supporting productive researchers and developing competitive graduate programming opportunities.
Project description:Scientific research funding is allocated largely through a system of soliciting and ranking competitive grant proposals. In these competitions, the proposals themselves are not the deliverables that the funder seeks, but instead are used by the funder to screen for the most promising research ideas. Consequently, some of the funding program's impact on science is squandered because applying researchers must spend time writing proposals instead of doing science. To what extent does the community's aggregate investment in proposal preparation negate the scientific impact of the funding program? Are there alternative mechanisms for awarding funds that advance science more efficiently? We use the economic theory of contests to analyze how efficiently grant proposal competitions advance science, and compare them with recently proposed, partially randomized alternatives such as lotteries. We find that the effort researchers waste in writing proposals may be comparable to the total scientific value of the research that the funding supports, especially when only a few proposals can be funded. Moreover, when professional pressures motivate investigators to seek funding for reasons that extend beyond the value of the proposed science (e.g., promotion, prestige), the entire program can actually hamper scientific progress when the number of awards is small. We suggest that lost efficiency may be restored either by partial lotteries for funding or by funding researchers based on past scientific success instead of proposals for future work.
Project description:Grantsmanship is an integral component of surviving and thriving in academic science, especially in the current funding climate. Therefore, any additional opportunities to write, read, and review grants during graduate school may have lasting benefits on one's career. We present here our experience with a small, student-run grant program at Georgetown University Medical Center. Founded in 2010, this program has several goals: 1) to give graduate students an opportunity to conduct small, independent research projects; 2) to encourage graduate students to write grants early and often; and 3) to give graduate students an opportunity to review grants. In the 3 yr since the program's start, 28 applications have been submitted, 13 of which were funded for a total of $40,000. From funded grants, students have produced abstracts and manuscripts, generated data to support subsequent grant proposals, and made new professional contacts with collaborators. Above and beyond financial support, this program provided both applicants and reviewers an opportunity to improve their writing skills, professional development, and understanding of the grants process, as reflected in the outcome measures presented. With a small commitment of time and funding, other institutions could implement a program like this to the benefit of their graduate students.
Project description:OBJECTIVES:To examine whether the gender of applicants and peer reviewers and other factors influence peer review of grant proposals submitted to a national funding agency. SETTING:Swiss National Science Foundation (SNSF). DESIGN:Cross-sectional analysis of peer review reports submitted from 2009 to 2016 using linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period. PARTICIPANTS:External peer reviewers. PRIMARY OUTCOME MEASURE:Overall score on a scale from 1 (worst) to 6 (best). RESULTS:Analyses included 38?250 reports on 12?294 grant applications from medicine, architecture, biology, chemistry, economics, engineering, geology, history, linguistics, mathematics, physics, psychology and sociology submitted by 26?829 unique peer reviewers. In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.18 points; 95%?CI 0.14 to 0.23), and male reviewers awarded higher scores than female reviewers (+0.11; 95%?CI 0.08 to 0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95%?CI 0.50 to 0.56), and reviewers from outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95%?CI 0.49 to 0.56). In multivariable analysis, differences between male and female applicants were attenuated (+0.08; 95%?CI 0.04 to 0.13) whereas results changed little for source of nomination and affiliation of reviewers. The gender difference increased after September 2011, when new evaluation forms were introduced (p=0.033 from test of interaction). CONCLUSIONS:Peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant's track record. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.
Project description:A classic thesis is that scientific achievement exhibits a "Matthew effect": Scientists who have previously been successful are more likely to succeed again, producing increasing distinction. We investigate to what extent the Matthew effect drives the allocation of research funds. To this end, we assembled a dataset containing all review scores and funding decisions of grant proposals submitted by recent PhDs in a €2 billion granting program. Analyses of review scores reveal that early funding success introduces a growing rift, with winners just above the funding threshold accumulating more than twice as much research funding (€180,000) during the following eight years as nonwinners just below it. We find no evidence that winners' improved funding chances in subsequent competitions are due to achievements enabled by the preceding grant, which suggests that early funding itself is an asset for acquiring later funding. Surprisingly, however, the emergent funding gap is partly created by applicants, who, after failing to win one grant, apply for another grant less often.
Project description:We investigated the association between a U.S. National Institutes of Health (NIH) R01 applicant's self-identified race or ethnicity and the probability of receiving an award by using data from the NIH IMPAC II grant database, the Thomson Reuters Web of Science, and other sources. Although proposals with strong priority scores were equally likely to be funded regardless of race, we find that Asians are 4 percentage points and black or African-American applicants are 13 percentage points less likely to receive NIH investigator-initiated research funding compared with whites. After controlling for the applicant's educational background, country of origin, training, previous research awards, publication record, and employer characteristics, we find that black applicants remain 10 percentage points less likely than whites to be awarded NIH research funding. Our results suggest some leverage points for policy intervention.
Project description:INTRODUCTION:Previous studies have examined the impact of Clinical and Translational Science Awards programs on other outcomes, but not on grant seeking. The authors examined the effects on grant seeking of the Michigan Institute for Clinical & Health Research (MICHR), a Clinical and Translational Science Awards institute at the University of Michigan. METHODS:We assessed over 63,000 grant proposals submitted at the University of Michigan in the years 2002-2012 using data from the university and MICHR's Tracking Metrics and Reporting System. We used a retrospective, observational study of the dynamics of grant-seeking success and award funding. Heckman selection models were run to assess MICHR's relationship with a proposal's success (selection), and subsequently the award's size (outcome). Models were run for all proposals and for clinical and translational research (CTR) proposals alone. Other covariates included proposal classification, type of grant award, academic unit, and year. RESULTS:MICHR had a positive and statistically significant relationship with success for both proposal types. For all grants, MICHR was associated with a 29.6% increase in award size. For CTR grants, MICHR had a statistically nonsignificant relationship with award size. CONCLUSIONS:MICHR's infrastructure, created to enable and enhance CTR, has also created positive spillovers for a broader spectrum of research and grant seeking.
Project description:Health services and policy research is the innovation engine of a health care system. In 2000, the Canadian Institutes of Health Research (CIHR) was formed to foster the growth of all sciences that could improve health care. We evaluated trends in health services and policy research funding, in addition to determinants of funding success.All applications submitted to CIHR strategic and open operating grant competitions between 2001 and 2011 were included in our analysis. Age, sex, size of research team, critical mass, season, year and research discipline were retrieved from application information. A cohort of 4725 applicants successfully funded between 2001 and 2005 were followed for 5 years to evaluate predictors of continuous funding. Multivariate generalized estimating equation logistic regression was used to estimate predictors of funding success and sustained funding.Between 2001 and 2011, 80?163 applications were submitted to open and strategic grant competitions. Over time, grant applications increased from 327 to 1137 per year, and annual funding increased from $12.6 to $48.0 million. Grant applications from young male researchers were more likely to be funded than those from female researchers (odds ratio [OR] 1.40, 95% confidence interval [CI] 1.01-1.95), as were applications from larger research teams and institutions with a large critical mass. Only 24.0% of scientists whose first funded grant was in health services and policy research had sustained 5-year funding, compared with 52.8% of biomedical scientists (OR 0.34, 95% CI 0.24-0.49).The CIHR has successfully increased the amount of health services and policy research in Canada. To enhance conditions for success, researchers should be encouraged to work in teams, request longer duration grants, resubmit unsuccessful applications and affiliate themselves with institutions with a greater critical mass.
Project description:One of the most frequently voiced criticisms of the peer review process is gender bias. In this study we evaluated the grant peer review process (external reviewers' ratings, and board of trustees' final decision: approval or no approval for funding) at the Austrian Science Fund with respect to gender. The data consisted of 8,496 research proposals (census) across all disciplines from 1999 to 2009, which were rated on a scale from 1 to 100 (poor to excellent) by 18,357 external reviewers in 23,977 reviews. In line with the current state of research, we found that the final decision was not associated with applicant's gender or with any correspondence between gender of applicants and reviewers. However, the decisions on the grant applications showed a robust female reviewer salience effect. The approval probability decreases (up to 10%), when there is parity or a majority of women in the group of reviewers. Our results confirm an overall gender null hypothesis for the peer review process of men's and women's grant applications in contrast to claims that women's grants are systematically downrated.
Project description:BACKGROUND:Although women at all career stages are more likely to leave academia than men, early-career women are a particularly high-risk group. Research supports that women are less likely than men to receive research funding; however, whether funding success rates vary based on research content is unknown. We addressed gender differences in funding success rates for applications directed to one or more of 13 institutes, representing research communities, over a 15-year period. METHODS AND FINDINGS:We retrospectively reviewed 55,700 grant and 4,087 personnel award applications submitted to the Canadian Institutes of Health Research. We analyzed application success rates according to gender and the primary institute selected by applicants, pooled gender differences in success rates using random effects models, and fitted Poisson regression models to assess the effects of gender, time, and institute. We noted variable success rates among grant applications directed to selected institutes and declining success rates over time. Women submitted 31.1% and 44.7% of grant and personnel award applications, respectively. In the pooled estimate, women had significantly lower grant success (risk ratio [RR] 0.89, 95% confidence interval [CI] 0.84-0.94; p < 0.001; absolute difference 3.2%) compared with men, with substantial heterogeneity (I2 = 58%). Compared with men, women who directed grants to the Institutes of Cancer Research (RR 0.86, 95% CI 0.78-0.96), Circulatory and Respiratory Health (RR 0.74, 95% CI 0.66-0.84), Health Services and Policy Research (RR 0.78, 95% CI 0.68-0.90), and Musculoskeletal Health and Arthritis (RR 0.80, 95% CI 0.69-0.93) were significantly less likely to be funded, and those who directed grants to the Institute of Aboriginal People's Health (RR 1.67, 95% CI 1.0-2.7) were more likely to be funded. Overall, women also had significantly lower personnel award success (RR 0.75, 95% CI 0.65-0.86; p < 0.001; absolute difference 6.6%). Regression modelling identified that the effect of gender on grant success rates differed by institute and not time. Study limitations include use of institutes as a surrogate identifier, variability in designation of primary institute, and lack of access to metrics reflecting applicants, coapplicants, peer reviewers, and the peer-review process. CONCLUSIONS:Gender disparity existed overall in grant and personnel award success rates, especially for grants directed to selected research communities. Funding agencies should monitor for gender differences in grant success rates overall and by research content.
Project description:We surveyed 113 astronomers and 82 psychologists active in applying for federally funded research on their grant-writing history between January, 2009 and November, 2012. We collected demographic data, effort levels, success rates, and perceived non-financial benefits from writing grant proposals. We find that the average proposal takes 116 PI hours and 55 CI hours to write; although time spent writing was not related to whether the grant was funded. Effort did translate into success, however, as academics who wrote more grants received more funding. Participants indicated modest non-monetary benefits from grant writing, with psychologists reporting a somewhat greater benefit overall than astronomers. These perceptions of non-financial benefits were unrelated to how many grants investigators applied for, the number of grants they received, or the amount of time they devoted to writing their proposals. We also explored the number of years an investigator can afford to apply unsuccessfully for research grants and our analyses suggest that funding rates below approximately 20%, commensurate with current NIH and NSF funding, are likely to drive at least half of the active researchers away from federally funded research. We conclude with recommendations and suggestions for individual investigators and for department heads.