Does Gender Matter in Grant Peer Review?: An Empirical Investigation Using the Example of the Austrian Science Fund.
ABSTRACT: One of the most frequently voiced criticisms of the peer review process is gender bias. In this study we evaluated the grant peer review process (external reviewers' ratings, and board of trustees' final decision: approval or no approval for funding) at the Austrian Science Fund with respect to gender. The data consisted of 8,496 research proposals (census) across all disciplines from 1999 to 2009, which were rated on a scale from 1 to 100 (poor to excellent) by 18,357 external reviewers in 23,977 reviews. In line with the current state of research, we found that the final decision was not associated with applicant's gender or with any correspondence between gender of applicants and reviewers. However, the decisions on the grant applications showed a robust female reviewer salience effect. The approval probability decreases (up to 10%), when there is parity or a majority of women in the group of reviewers. Our results confirm an overall gender null hypothesis for the peer review process of men's and women's grant applications in contrast to claims that women's grants are systematically downrated.
Project description:OBJECTIVES:To examine whether the gender of applicants and peer reviewers and other factors influence peer review of grant proposals submitted to a national funding agency. SETTING:Swiss National Science Foundation (SNSF). DESIGN:Cross-sectional analysis of peer review reports submitted from 2009 to 2016 using linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period. PARTICIPANTS:External peer reviewers. PRIMARY OUTCOME MEASURE:Overall score on a scale from 1 (worst) to 6 (best). RESULTS:Analyses included 38?250 reports on 12?294 grant applications from medicine, architecture, biology, chemistry, economics, engineering, geology, history, linguistics, mathematics, physics, psychology and sociology submitted by 26?829 unique peer reviewers. In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.18 points; 95%?CI 0.14 to 0.23), and male reviewers awarded higher scores than female reviewers (+0.11; 95%?CI 0.08 to 0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95%?CI 0.50 to 0.56), and reviewers from outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95%?CI 0.49 to 0.56). In multivariable analysis, differences between male and female applicants were attenuated (+0.08; 95%?CI 0.04 to 0.13) whereas results changed little for source of nomination and affiliation of reviewers. The gender difference increased after September 2011, when new evaluation forms were introduced (p=0.033 from test of interaction). CONCLUSIONS:Peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant's track record. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.
Project description:Obtaining grant funding from the National Institutes of Health (NIH) is increasingly competitive, as funding success rates have declined over the past decade. To allocate relatively scarce funds, scientific peer reviewers must differentiate the very best applications from comparatively weaker ones. Despite the importance of this determination, little research has explored how reviewers assign ratings to the applications they review and whether there is consistency in the reviewers' evaluation of the same application. Replicating all aspects of the NIH peer-review process, we examined 43 individual reviewers' ratings and written critiques of the same group of 25 NIH grant applications. Results showed no agreement among reviewers regarding the quality of the applications in either their qualitative or quantitative evaluations. Although all reviewers received the same instructions on how to rate applications and format their written critiques, we also found no agreement in how reviewers "translated" a given number of strengths and weaknesses into a numeric rating. It appeared that the outcome of the grant review depended more on the reviewer to whom the grant was assigned than the research proposed in the grant. This research replicates the NIH peer-review process to examine in detail the qualitative and quantitative judgments of different reviewers examining the same application, and our results have broad relevance for scientific grant peer review.
Project description:In scientific grant peer review, groups of expert scientists meet to engage in the collaborative decision-making task of evaluating and scoring grant applications. Prior research on grant peer review has established that inter-reviewer reliability is typically poor. In the current study, experienced reviewers for the National Institutes of Health (NIH) were recruited to participate in one of four constructed peer review panel meetings. Each panel discussed and scored the same pool of recently reviewed NIH grant applications. We examined the degree of intra-panel variability in panels' scores of the applications before versus after collaborative discussion, and the degree of inter-panel variability. We also analyzed videotapes of reviewers' interactions for instances of one particular form of discourse-Score Calibration Talk-as one factor influencing the variability we observe. Results suggest that although reviewers within a single panel agree more following collaborative discussion, different panels agree less after discussion, and Score Calibration Talk plays a pivotal role in scoring variability during peer review. We discuss implications of this variability for the scientific peer review process.
Project description:<h4>Objective</h4>To review the literature on strategies implemented or identified to prevent or reduce gender bias in peer review of research grants.<h4>Methods</h4>Studies of any type of qualitative or quantitative design examining interventions to reduce or prevent gender bias during the peer review of health-related research grants were included. Electronic databases including MEDLINE, EMBASE, Education Resources Information Center (ERIC), PsycINFO, Joanna Briggs, the Cochrane Library, Evidence Based Medicine (EBM) Reviews, and the Campbell Library were searched from 2005 to April 2016. A search for grey (i.e., difficult to locate or unpublished) literature was conducted and experts in the field were consulted to identify additional potentially relevant articles. Two individuals screened titles and abstracts, full-text articles, and abstracted data with discrepancies resolved by a third person consistently.<h4>Results</h4>After screening 5524 citations and 170 full-text articles, one article evaluating gender-blinding of grant applications using an uncontrolled before-after study design was included. In this study, 891 applications for long-term fellowships in 2006 were included and 47% of the applicants were women. These were scored by 13 peer reviewers (38% were women). The intervention included eliminating references to gender from the applications, letters of recommendations, and interview reports that were sent to the committee members for evaluation. The proportion of successful applications led by women did not change with gender-blinding, although the number of successful applications that were led by men increased slightly.<h4>Conclusions</h4>There is limited research on interventions to mitigate gender bias in the peer review of grants. Only one study was identified and no difference in the proportion of women who were successful in receiving grant funding was observed. Our results suggest that interventions to prevent gender bias should be adapted and tested in the context of grant peer review to determine if they will have an impact.
Project description:<h4>Background</h4>Peer review is the most widely used method for evaluating grant applications in clinical research. Criticisms of peer review include lack of equity, suspicion of biases, and conflicts of interest (CoI). CoIs raise questions of fairness, transparency, and trust in grant allocation. Few observational studies have assessed these issues. We report the results of a qualitative study on reviewers' and applicants' perceptions and experiences of CoIs in reviews of French academic grant applications.<h4>Methodology and principal findings</h4>We designed a qualitative study using semi-structured interviews and direct observation. We asked members of assessment panels, external reviewers, and applicants to participate in semi-structured interviews. Two independent researchers conducted in-depth reviews and line-by-line coding of all transcribed interviews, which were also subjected to Tropes® software text analysis, to detect and qualify themes associated with CoIs. Most participants (73/98) spontaneously reported that non-financial CoIs predominated over financial CoIs. Non-financial CoIs mainly involved rivalry among disciplines, cronyism, and geographic and academic biases. However, none of the participants challenged the validity of peer review. Reviewers who felt they might be affected by CoIs said they reacted in a variety of ways: routine refusal to review, routine attempt to conduct an impartial review, or decision on a case-by-case basis. Multiple means of managing non-financial CoIs were suggested, including increased transparency throughout the review process, with public disclosure of non-financial CoIs, and careful selection of independent reviewers, including foreign experts and methodologists.<h4>Conclusions</h4>Our study underscores the importance of considering non-financial CoIs when reviewing research grant applications, in addition to financial CoIs. Specific measures are needed to prevent a negative impact of non-financial CoIs on the fairness of resource allocation. Whether and how public disclosure of non-financial CoIs should be accomplished remains debatable.
Project description:BACKGROUND: Peer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers' scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria. METHODS AND FINDINGS: We first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest. CONCLUSIONS: Variability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.
Project description:Grant-writing and grant-getting are key to success in many academic disciplines, but research points to gender gaps in both, especially as careers progress. Using a sample of National Institutes of Health (NIH) K-Awardees-Principal Investigators of Mentored Career Development Awards-we examined gender and race effects in response to imagined negative grant reviews that emphasized either promise or inadequacy. Women translated both forms of feedback into worse NIH priority scores than did men and showed reduced motivation to reapply for funding following the review highlighting inadequacy. Translation of feedback mediated the effects of gender on motivation, changing one's research focus, and advice-seeking. Race effects were less consistent, and race did not moderate effects of gender. We suggest that gender bias in grant reviews (i.e., greater likelihood of highlighting inadequacy in reviews of women's grants), along with gender differences in responsiveness to feedback, may contribute to women's underrepresentation in academic medicine.
Project description:BACKGROUND:Although women at all career stages are more likely to leave academia than men, early-career women are a particularly high-risk group. Research supports that women are less likely than men to receive research funding; however, whether funding success rates vary based on research content is unknown. We addressed gender differences in funding success rates for applications directed to one or more of 13 institutes, representing research communities, over a 15-year period. METHODS AND FINDINGS:We retrospectively reviewed 55,700 grant and 4,087 personnel award applications submitted to the Canadian Institutes of Health Research. We analyzed application success rates according to gender and the primary institute selected by applicants, pooled gender differences in success rates using random effects models, and fitted Poisson regression models to assess the effects of gender, time, and institute. We noted variable success rates among grant applications directed to selected institutes and declining success rates over time. Women submitted 31.1% and 44.7% of grant and personnel award applications, respectively. In the pooled estimate, women had significantly lower grant success (risk ratio [RR] 0.89, 95% confidence interval [CI] 0.84-0.94; p < 0.001; absolute difference 3.2%) compared with men, with substantial heterogeneity (I2 = 58%). Compared with men, women who directed grants to the Institutes of Cancer Research (RR 0.86, 95% CI 0.78-0.96), Circulatory and Respiratory Health (RR 0.74, 95% CI 0.66-0.84), Health Services and Policy Research (RR 0.78, 95% CI 0.68-0.90), and Musculoskeletal Health and Arthritis (RR 0.80, 95% CI 0.69-0.93) were significantly less likely to be funded, and those who directed grants to the Institute of Aboriginal People's Health (RR 1.67, 95% CI 1.0-2.7) were more likely to be funded. Overall, women also had significantly lower personnel award success (RR 0.75, 95% CI 0.65-0.86; p < 0.001; absolute difference 6.6%). Regression modelling identified that the effect of gender on grant success rates differed by institute and not time. Study limitations include use of institutes as a surrogate identifier, variability in designation of primary institute, and lack of access to metrics reflecting applicants, coapplicants, peer reviewers, and the peer-review process. CONCLUSIONS:Gender disparity existed overall in grant and personnel award success rates, especially for grants directed to selected research communities. Funding agencies should monitor for gender differences in grant success rates overall and by research content.
Project description:Most funding agencies rely on peer review to evaluate grant applications and proposals, but research into the use of this process by funding agencies has been limited. Here we explore if two changes to the organization of peer review for proposals submitted to various funding actions by the European Union has an influence on the outcome of the peer review process. Based on an analysis of more than 75,000 applications to three actions of the Marie Curie programme over a period of 12 years, we find that the changes - a reduction in the number of evaluation criteria used by reviewers and a move from in-person to virtual meetings - had little impact on the outcome of the peer review process. Our results indicate that other factors, such as the type of grant or area of research, have a larger impact on the outcome.
Project description:Peer review is the cornerstone of scholarly publishing and it is essential that peer reviewers are appointed on the basis of their expertise alone. However, it is difficult to check for any bias in the peer-review process because the identity of peer reviewers generally remains confidential. Here, using public information about the identities of 9000 editors and 43000 reviewers from the Frontiers series of journals, we show that women are underrepresented in the peer-review process, that editors of both genders operate with substantial same-gender preference (homophily), and that the mechanisms of this homophily are gender-dependent. We also show that homophily will persist even if numerical parity between genders is reached, highlighting the need for increased efforts to combat subtler forms of gender bias in scholarly publishing.