Dataset Information

DORA-compliant measures of research quality and impact to assess the performance of researchers in biomedical institutions: Review of published research, international best practice and Delphi survey.

ABSTRACT:

Objective

The San Francisco Declaration on Research Assessment (DORA) advocates for assessing biomedical research quality and impact, yet academic organizations continue to employ traditional measures such as Journal Impact Factor. We aimed to identify and prioritize measures for assessing research quality and impact.

Methods

We conducted a review of published and grey literature to identify measures of research quality and impact, which we included in an online survey. We assembled a panel of researchers and research leaders, and conducted a two-round Delphi survey to prioritize measures rated as high (rated 6 or 7 by ≥ 80% of respondents) or moderate (rated 6 or 7 by ≥ 50% of respondents) importance.

Results

We identified 50 measures organized in 8 domains: relevance of the research program, challenges to research program, or productivity, team/open science, funding, innovations, publications, other dissemination, and impact. Rating of measures by 44 panelists (60%) in Round One and 24 (55%) in Round Two of a Delphi survey resulted in consensus on the high importance of 5 measures: research advances existing knowledge, research plan is innovative, an independent body of research (or fundamental role) supported by peer-reviewed research funding, research outputs relevant to discipline, and quality of the content of publications. Five measures achieved consensus on moderate importance: challenges to research productivity, potential to improve health or healthcare, team science, collaboration, and recognition by professional societies or academic bodies. There was high congruence between researchers and research leaders across disciplines.

Conclusions

Our work contributes to the field by identifying 10 DORA-compliant measures of research quality and impact, a more comprehensive and explicit set of measures than prior efforts. Research is needed to identify strategies to overcome barriers of use of DORA-compliant measures, and to "de-implement" traditional measures that do not uphold DORA principles yet are still in use.

SUBMITTER: Gagliardi AR

PROVIDER: S-EPMC10180594 | biostudies-literature | 2023

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

DORA-compliant measures of research quality and impact to assess the performance of researchers in biomedical institutions: Review of published research, international best practice and Delphi survey.

Gagliardi Anna R AR Chen Rob H C RHC Boury Himani H Albert Mathieu M Chow James J DaCosta Ralph S RS Hoffman Michael M Keshavarz Behrang B Kontos Pia P Liu Jenny J McAndrews Mary Pat MP Protze Stephanie S

PloS one 20230512 5

<h4>Objective</h4>The San Francisco Declaration on Research Assessment (DORA) advocates for assessing biomedical research quality and impact, yet academic organizations continue to employ traditional measures such as Journal Impact Factor. We aimed to identify and prioritize measures for assessing research quality and impact.<h4>Methods</h4>We conducted a review of published and grey literature to identify measures of research quality and impact, which we included in an online survey. We assembl ...[more]

PMID: 37172046

Similar Datasets

Project description:We conducted an international cross-sectional survey of biomedical researchers' perspectives on the reproducibility of research. This study builds on a widely cited 2016 survey on reproducibility and provides a biomedical-specific and contemporary perspective on reproducibility. To sample the community, we randomly selected 400 journals indexed in MEDLINE, from which we extracted the author names and emails from all articles published between October 1, 2020 and October 1, 2021. We invited participants to complete an anonymous online survey which collected basic demographic information, perceptions about a reproducibility crisis, perceived causes of irreproducibility of research results, experience conducting reproducibility studies, and knowledge of funding and training for research on reproducibility. A total of 1,924 participants accessed our survey, of which 1,630 provided useable responses (response rate 7% of 23,234). Key findings include that 72% of participants agreed there was a reproducibility crisis in biomedicine, with 27% of participants indicating the crisis was "significant." The leading perceived cause of irreproducibility was a "pressure to publish" with 62% of participants indicating it "always" or "very often" contributes. About half of the participants (54%) had run a replication of their own previously published study while slightly more (57%) had run a replication of another researcher's study. Just 16% of participants indicated their institution had established procedures to enhance the reproducibility of biomedical research and 67% felt their institution valued new research over replication studies. Participants also reported few opportunities to obtain funding to attempt to reproduce a study and 83% perceived it would be harder to do so than to get funding to do a novel study. Our results may be used to guide training and interventions to improve research reproducibility and to monitor rates of reproducibility over time. The findings are also relevant to policy makers and academic leadership looking to create incentives and research cultures that support reproducibility and value research quality.

Project description:BackgroundKnowledge translation (KT) is a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically sound application of knowledge to yield beneficial outcomes for society. Effective KT requires researchers to play an active role in promoting evidence uptake. This paper presents a systematised review of evidence on low- and middle-income country (LMIC) researchers' KT capacity, practice and interventions for enhancing their KT practice (support) with the aim of identifying gaps and informing future research and interventions.MethodsAn electronic search for peer-reviewed publications focusing on LMIC researchers' KT capacity, practice and support across all academic fields, authored in English and from the earliest records available to February 2019, was conducted using PubMed and Scopus. Selected studies were appraised using the Mixed Methods Appraisal Tool, data pertaining to publication characteristics and study design extracted, and an a priori thematic analysis of reported research findings completed.ResultsThe search resulted in 334 screened articles, of which 66 met the inclusion criteria. Most (n = 43) of the articles presented original research findings, 22 were commentaries and 1 was a structured review; 47 articles reported on researchers' KT practice, 12 assessed the KT capacity of researchers or academic/research institutions and 9 reported on KT support for researchers. More than half (59%) of the articles focused on sub-Saharan Africa and the majority (91%) on health research. Most of the primary studies used the case study design (41%). The findings suggest that LMIC researchers rarely conduct KT and face a range of barriers at individual and institutional levels that limit their KT practice, including inadequate KT knowledge and skills, particularly for communicating research and interacting with research end-users, insufficient funding, and inadequate institutional guidelines, structures and incentives promoting KT practice. Furthermore, the evidence-base on effective interventions for enhancing LMIC researchers' KT practice is insufficient and largely of weak quality.ConclusionsMore high-quality research on researchers' KT capacity, practice and effective KT capacity strengthening interventions is needed. Study designs that extend beyond case studies and descriptive studies are recommended, including better designed evaluation studies, e.g. use of realist approaches, pragmatic trials, impact evaluations, implementation research and participatory action research.

Project description:Background: The challenge of research funding constraints has brought to bear enormous pressure on researchers. Research productivity is relevant to prestige and career progression of academic staff. However, this study aimed to explore significant challenges associated with researchers' productivity and the impact of non-funding of research in Nigerian research and tertiary institutions. Methods: This study adopted a qualitative exploratory design involving academics at various research and tertiary institutions across the six geographical regions in Nigeria. A semi-structured questionnaire was distributed electronically to all participants who consented to take part in this study. Exactly 4,159 questionnaires were administered and 2,350 were completely filled and returned. Pearson correlation matrices with logistic regression were used for data analysis and are presented in frequencies and percentages. Results: On challenges faced by respondents, 42.98% reported a lack of research funding, 17.11% mentioned brain drain challenge while 8.85% indicated a lack of motivation. Of the 23,927 publications reported, the number of those in sciences, engineering, and medical sciences averaged 9.6, 11.5, and 9.5 respectively. The average number of publications by women (10.8) was more than by men (9.7). Lecturers had the highest average research publication number (11.8) followed by researchers (10.2) and others (3.9). Men had the highest (11.9) average number of conferences compared to women (9.2). Participants in engineering had an average number of 13.8 conferences per respondents followed by those in education (11.2), sciences (11.1), and 10.9 for those in agricultural sciences. The result revealed a negative significant correlation between research publication and academic qualification at p < 0.01. Positive significant correlation was observed between research productivity and discipline at p < 0.05. Findings show that the combined influence of the independent variables on research productivity was significant using linear regression analysis. Conclusions: The failure to prioritize research has resulted in underdevelopment in Nigeria. It is therefore imperative that the federal government prioritize research and establish a functional Special Research Trust Fund to oversee research funding in Nigeria.

Dataset Information

DORA-compliant measures of research quality and impact to assess the performance of researchers in biomedical institutions: Review of published research, international best practice and Delphi survey.

Objective

Methods

Results

Conclusions

Publications

DORA-compliant measures of research quality and impact to assess the performance of researchers in biomedical institutions: Review of published research, international best practice and Delphi survey.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets