Development and evaluation of a quality score for abstracts.
ABSTRACT: BACKGROUND:The evaluation of abstracts for scientific meetings has been shown to suffer from poor inter observer reliability. A measure was developed to assess the formal quality of abstract submissions in a standardized way. METHODS:Item selection was based on scoring systems for full reports, taking into account published guidelines for structured abstracts. Interrater agreement was examined using a random sample of submissions to the American Gastroenterological Association, stratified for research type (n = 100, 1992-1995). For construct validity, the association of formal quality with acceptance for presentation was examined. A questionnaire to expert reviewers evaluated sensibility items, such as ease of use and comprehensiveness. RESULTS:The index comprised 19 items. The summary quality scores showed good interrater agreement (intra class coefficient 0.60 - 0.81). Good abstract quality was associated with abstract acceptance for presentation at the meeting. The instrument was found to be acceptable by expert reviewers. CONCLUSION:A quality index was developed for the evaluation of scientific meeting abstracts which was shown to be reliable, valid and useful.
Project description:BACKGROUND:It has been estimated that about 45% of abstracts that are accepted for presentation at biomedical meetings will subsequently be published in full. The acceptance of abstracts at meetings and their fate after initial rejection are less well understood. We set out to estimate the proportion of abstracts submitted to meetings that are eventually published as full reports, and to explore factors that are associated with meeting acceptance and successful publication. METHODS:Studies analysing acceptance of abstracts at biomedical meetings or their subsequent full publication were searched in MEDLINE, OLDMEDLINE, EMBASE, Cochrane Library, CINAHL, BIOSIS, Science Citation Index Expanded, and by hand searching of bibliographies and proceedings. We estimated rates of abstract acceptance and of subsequent full publication, and identified abstract and meeting characteristics associated with acceptance and publication, using logistic regression analysis, survival-type analysis, and meta-analysis. RESULTS:Analysed meetings were held between 1957 and 1999. Of 14945 abstracts that were submitted to 43 meetings, 46% were accepted. The rate of full publication was studied with 19123 abstracts that were presented at 234 meetings. Using survival-type analysis, we estimated that 27% were published after two, 41% after four, and 44% after six years. Of 2412 abstracts that were rejected at 24 meetings, 27% were published despite rejection. Factors associated with both abstract acceptance and subsequent publication were basic science and positive study outcome. Large meetings and those held outside the US were more likely to accept abstracts. Abstracts were more likely to be published subsequently if presented either orally, at small meetings, or at a US meeting. Abstract acceptance itself was strongly associated with full publication. CONCLUSIONS:About one third of abstracts submitted to biomedical meetings were published as full reports. Acceptance at meetings and publication were associated with specific characteristics of abstracts and meetings.
Project description:CytoJournal, with its continued contribution of scientific cytopathology literature to the public domain under open access (OA) charter, thanks its dedicated peer reviewers for devoting significant efforts, time, and resources during 2011. The abstracts of poster-platform submissions to the 59(th) Annual Scientific Meeting (November 2011) of the American Society of Cytopathology (ASC) in Baltimore, MD, USA, were peer reviewed by the ASC Scientific Program Committee.
Project description:Peer review may be "single-blind," in which reviewers are aware of the names and affiliations of paper authors, or "double-blind," in which this information is hidden. Noting that computer science research often appears first or exclusively in peer-reviewed conferences rather than journals, we study these two reviewing models in the context of the 10th Association for Computing Machinery International Conference on Web Search and Data Mining, a highly selective venue (15.6% acceptance rate) in which expert committee members review full-length submissions for acceptance. We present a controlled experiment in which four committee members review each paper. Two of these four reviewers are drawn from a pool of committee members with access to author information; the other two are drawn from a disjoint pool without such access. This information asymmetry persists through the process of bidding for papers, reviewing papers, and entering scores. Reviewers in the single-blind condition typically bid for 22% fewer papers and preferentially bid for papers from top universities and companies. Once papers are allocated to reviewers, single-blind reviewers are significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors, top universities, and top companies. The estimated odds multipliers are tangible, at 1.63, 1.58, and 2.10, respectively.
Project description:<h4>Background</h4>Scientists communicate progress and exchange information via publication and presentation at scientific meetings. We previously showed that text similarity analysis applied to Medline can identify and quantify plagiarism and duplicate publications in peer-reviewed biomedical journals. In the present study, we applied the same analysis to a large sample of conference abstracts.<h4>Methods</h4>We downloaded 144,149 abstracts from 207 national and international meetings of 63 biomedical conferences. Pairwise comparisons were made using eTBLAST: a text similarity engine. A domain expert then reviewed random samples of highly similar abstracts (1500 total) to estimate the extent of text overlap and possible plagiarism.<h4>Results</h4>Our main findings indicate that the vast majority of textual overlap occurred within the same meeting (2%) and between meetings of the same conference (3%), both of which were significantly higher than instances of plagiarism, which occurred in less than .5% of abstracts.<h4>Conclusions</h4>This analysis indicates that textual overlap in abstracts of papers presented at scientific meetings is one-tenth that of peer-reviewed publications, yet the plagiarism rate is approximately the same as previously measured in peer-reviewed publications. This latter finding underscores a need for monitoring scientific meeting submissions - as is now done when submitting manuscripts to peer-reviewed journals - to improve the integrity of scientific communications.
Project description:<h4>Introduction</h4>Nothing has been published to describe the practices of medical societies in choosing abstracts for presentations at their annual meetings. We surveyed medical societies to determine their practices, and also present a theoretical analysis of the topic.<h4>Methods</h4>We contacted a convenience sample of large U.S. medical conferences, and determined their approach to choosing abstracts. We obtained information from web sites, telephone, and email. Our theoretical analysis compares values-based and empirical approaches for scoring system development.<h4>Results</h4>We contacted 32 societies and obtained data on 28 (response rate 88%). We excluded one upon learning that research was not presented at its annual meeting, leaving 27 for analysis. Only 2 (7%) made their abstract scoring process available to submitters. Reviews were blinded in most societies (21;78%), and all but one asked reviewers to recuse themselves for conflict of interest (96%). All required ?3 reviewers. Of the 24 providing information on how scores were generated, 21 (88%) reported using a single gestalt score, and three used a combined score created from pooled domain-specific sub-scores. We present a framework for societies to use in choosing abstracts, and demonstrate its application in the development of a new scoring system.<h4>Conclusions</h4>Most medical societies use subjective, gestalt methods to select research for presentation at their annual meetings and do not disclose to submitters the details of how abstracts are chosen. We present a new scoring system that is transparent to submitters and reviewers alike with an accompanying statement of values and ground rules. We discuss the challenges faced in selecting abstracts for a large scientific meeting and share the values and practical considerations that undergird the new system.
Project description:IMPORTANCE:Conference abstracts present information that helps clinicians and researchers to decide whether to attend a presentation. They also provide a source of unpublished research that could potentially be included in systematic reviews. We systematically assessed whether conference abstracts of studies that evaluated the accuracy of a diagnostic test were sufficiently informative. OBSERVATIONS:We identified all abstracts describing work presented at the 2010 Annual Meeting of the Association for Research in Vision and Ophthalmology. Abstracts were eligible if they included a measure of diagnostic accuracy, such as sensitivity, specificity, or likelihood ratios. Two independent reviewers evaluated each abstract using a list of 21 items, selected from published guidance for adequate reporting. A total of 126 of 6310 abstracts presented were eligible. Only a minority reported inclusion criteria (5%), clinical setting (24%), patient sampling (10%), reference standard (48%), whether test readers were masked (7%), 2?×?2 tables (16%), and confidence intervals around accuracy estimates (16%). The mean number of items reported was 8.9 of 21 (SD, 2.1; range, 4-17). CONCLUSIONS AND RELEVANCE:Crucial information about study methods and results is often missing in abstracts of diagnostic studies presented at the Association for Research in Vision and Ophthalmology Annual Meeting, making it difficult to assess risk for bias and applicability to specific clinical settings.
Project description:BACKGROUND: Head & Face Medicine (HFM) was launched in August 2005 to provide multidisciplinary science in the field of head and face disorders with an open access and open peer review publication platform. The objective of this study is to evaluate the characteristics of submissions, the effectiveness of open peer reviewing, and factors biasing the acceptance or rejection of submitted manuscripts. METHODS: A 1-year period of submissions and all concomitant journal operations were retrospectively analyzed. The analysis included submission rate, reviewer rate, acceptance rate, article type, and differences in duration for peer reviewing, final decision, publishing, and PubMed inclusion. Statistical analysis included Mann-Whitney U test, Chi-square test, regression analysis, and binary logistic regression. RESULTS: HFM received 126 articles (10.5 articles/month) for consideration in the first year. Submissions have been increasing, but not significantly over time. Peer reviewing was completed for 82 articles and resulted in an acceptance rate of 48.8%. In total, 431 peer reviewers were invited (5.3/manuscript), of which 40.4% agreed to review. The mean peer review time was 37.8 days. The mean time between submission and acceptance (including time for revision) was 95.9 days. Accepted papers were published on average 99.3 days after submission. The mean time between manuscript submission and PubMed inclusion was 101.3 days. The main article types submitted to HFM were original research, reviews, and case reports. The article type had no influence on rejection or acceptance. The variable 'number of invited reviewers' was the only significant (p < 0.05) predictor for rejection of manuscripts. CONCLUSION: The positive trend in submissions confirms the need for publication platforms for multidisciplinary science. HFM's peer review time comes in shorter than the 6-weeks turnaround time the Editors set themselves as the maximum. Rejection of manuscripts was associated with the number of invited reviewers. None of the other parameters tested had any effect on the final decision. Thus, HFM's ethical policy, which is based on Open Access, Open Peer, and transparency of journal operations, is free of 'editorial bias' in accepting manuscripts. ORIGINAL DATA: Provided as a downloadable tab-delimited text file (URL and variable code available under section 'additional files').
Project description:Background:Descriptive studies examining publication rates and citation counts demonstrate a geographic skew toward high-income countries (HIC), and research from low- or middle-income countries (LMICs) is generally underrepresented. This has been suggested to be due in part to reviewers' and editors' preference toward HIC sources; however, in the absence of controlled studies, it is impossible to assert whether there is bias or whether variations in the quality or relevance of the articles being reviewed explains the geographic divide. This study synthesizes the evidence from randomized and controlled studies that explore geographic bias in the peer review process. Methods:A systematic review was conducted to identify research studies that explicitly explore the role of geographic bias in the assessment of the quality of research articles. Only randomized and controlled studies were included in the review. Five databases were searched to locate relevant articles. A narrative synthesis of included articles was performed to identify common findings. Results:The systematic literature search yielded 3501 titles from which 12 full texts were reviewed, and a further eight were identified through searching reference lists of the full texts. Of these articles, only three were randomized and controlled studies that examined variants of geographic bias. One study found that abstracts attributed to HIC sources elicited a higher review score regarding relevance of the research and likelihood to recommend the research to a colleague, than did abstracts attributed to LIC sources. Another study found that the predicted odds of acceptance for a submission to a computer science conference were statistically significantly higher for submissions from a "Top University." Two of the studies showed the presence of geographic bias between articles from "high" or "low" prestige institutions. Conclusions:Two of the three included studies identified that geographic bias in some form was impacting on peer review; however, further robust, experimental evidence is needed to adequately inform practice surrounding this topic. Reviewers and researchers should nonetheless be aware of whether author and institutional characteristics are interfering in their judgement of research.
Project description:OBJECTIVES:To evaluate the reporting quality of randomised controlled trial (RCT) abstracts presented at a leading international conference in sleep medicine (the SLEEP Annual Meeting), and to investigate the association between potential predictors and the reporting quality of trial abstracts in this field. DESIGN:Cross-sectional, research on research study. METHODS:A handsearch of the 2016-2018 SLEEP Annual Meeting abstract books was carried out to identify abstracts describing RCTs. Quality of reporting was assessed with the original 17-item CONSORT for Abstracts checklist. Univariable and multivariable linear regression analyses were performed to identify significant predictors of reporting quality. In addition, risk ratios were used to analyse the adequate reporting rate of each quality item by type of intervention and funding status. PRIMARY AND SECONDARY OUTCOME MEASURES:The overall quality score (OQS, range 0-17) in accordance with the CONSORT for Abstracts checklist (primary outcome), and the adequate reporting rate of each checklist item (secondary outcome). RESULTS:A total of 176 RCT abstracts were included and assessed. The mean OQS was 5.53 (95% CI 5.30 to 5.76). Only three quality items (objective, conclusions and funding) were adequately reported in most abstracts (>75%). None of the abstracts adequately reported authors, randomisation or outcome in the results section. According to the multivariable analysis, pharmacological interventions (p=0.018) and funding from the industry (p=0.025) were significantly associated with better reporting quality. CONCLUSIONS:The reporting quality of RCT abstracts presented at SLEEP Annual Meetings was suboptimal. Pharmacological intervention and funding from industry were significant predictors of better reporting quality. Joint efforts by authors and conference committees are needed to enhance the reporting quality of RCT abstracts presented at sleep medicine conferences, and thereby reduce relevant research waste in this field.
Project description:BACKGROUND:Web applications that employ natural language processing technologies to support systematic reviewers during abstract screening have become more common. The goal of our project was to conduct a case study to explore a screening approach that temporarily replaces a human screener with a semi-automated screening tool. METHODS:We evaluated the accuracy of the approach using DistillerAI as a semi-automated screening tool. A published comparative effectiveness review served as the reference standard. Five teams of professional systematic reviewers screened the same 2472 abstracts in parallel. Each team trained DistillerAI with 300 randomly selected abstracts that the team screened dually. For all remaining abstracts, DistillerAI replaced one human screener and provided predictions about the relevance of records. A single reviewer also screened all remaining abstracts. A second human screener resolved conflicts between the single reviewer and DistillerAI. We compared the decisions of the machine-assisted approach, single-reviewer screening, and screening with DistillerAI alone against the reference standard. RESULTS:The combined sensitivity of the machine-assisted screening approach across the five screening teams was 78% (95% confidence interval [CI], 66 to 90%), and the combined specificity was 95% (95% CI, 92 to 97%). By comparison, the sensitivity of single-reviewer screening was similar (78%; 95% CI, 66 to 89%); however, the sensitivity of DistillerAI alone was substantially worse (14%; 95% CI, 0 to 31%) than that of the machine-assisted screening approach. Specificities for single-reviewer screening and DistillerAI were 94% (95% CI, 91 to 97%) and 98% (95% CI, 97 to 100%), respectively. Machine-assisted screening and single-reviewer screening had similar areas under the curve (0.87 and 0.86, respectively); by contrast, the area under the curve for DistillerAI alone was just slightly better than chance (0.56). The interrater agreement between human screeners and DistillerAI with a prevalence-adjusted kappa was 0.85 (95% CI, 0.84 to 0.86%). CONCLUSIONS:The accuracy of DistillerAI is not yet adequate to replace a human screener temporarily during abstract screening for systematic reviews. Rapid reviews, which do not require detecting the totality of the relevant evidence, may find semi-automation tools to have greater utility than traditional systematic reviews.