In Pursuit of Greater Reproducibility and Credibility of Early Clinical Biomarker Research.
Ontology highlight
ABSTRACT:
SUBMITTER: McShane LM
PROVIDER: S-EPMC5355975 | biostudies-literature | 2017 Mar
REPOSITORIES: biostudies-literature
Publications
In Pursuit of Greater Reproducibility and Credibility of Early Clinical Biomarker Research.
Clinical and translational science 20170306 2
Similar Datasets
Project description:High-profile reports of detrimental scientific practices leading to retractions in the scientific literature contribute to lack of trust in scientific experts. Although the bulk of these have been in the literature of other disciplines, environmental toxicology and chemistry are not free from problems. While we believe that egregious misconduct such as fraud, fabrication of data, or plagiarism is rare, scientific integrity is much broader than the absence of misconduct. We are more concerned with more commonly encountered and nuanced issues such as poor reliability and bias. We review a range of topics including conflicts of interests, competing interests, some particularly challenging situations, reproducibility, bias, and other attributes of ecotoxicological studies that enhance or detract from scientific credibility. Our vision of scientific integrity encourages a self-correcting culture that promotes scientific rigor, relevant reproducible research, transparency in competing interests, methods and results, and education. Integr Environ Assess Manag 2019;00:000-000. © 2019 SETAC.
| S-EPMC7313240 | biostudies-literature
Project description:Modeling and simulation in computational neuroscience is currently a research enterprise to better understand neural systems. It is not yet directly applicable to the problems of patients with brain disease. To be used for clinical applications, there must not only be considerable progress in the field but also a concerted effort to use best practices in order to demonstrate model credibility to regulatory bodies, to clinics and hospitals, to doctors, and to patients. In doing this for neuroscience, we can learn lessons from long-standing practices in other areas of simulation (aircraft, computer chips), from software engineering, and from other biomedical disciplines. In this manuscript, we introduce some basic concepts that will be important in the development of credible clinical neuroscience models: reproducibility and replicability; verification and validation; model configuration; and procedures and processes for credible mechanistic multiscale modeling. We also discuss how garnering strong community involvement can promote model credibility. Finally, in addition to direct usage with patients, we note the potential for simulation usage in the area of Simulation-Based Medical Education, an area which to date has been primarily reliant on physical models (mannequins) and scenario-based simulations rather than on numerical simulations.
| S-EPMC5911506 | biostudies-literature
Project description:During the COVID-19 pandemic, mathematical modeling of disease transmission has become a cornerstone of key state decisions. To advance the state-of-the-art host viral modeling to handle future pandemics, many scientists working on related issues assembled to discuss the topics. These discussions exposed the reproducibility crisis that leads to inability to reuse and integrate models. This document summarizes these discussions, presents difficulties, and mentions existing efforts towards future solutions that will allow future model utility and integration. We argue that without addressing these challenges, scientists will have diminished ability to build, disseminate, and implement high-impact multi-scale modeling that is needed to understand the health crises we face.
| S-EPMC10002468 | biostudies-literature
Project description:BackgroundThe ability to reproduce experiments is a defining principle of science. Reproducibility of clinical research has received relatively little scientific attention. However, it is important as it may inform clinical practice, research agendas, and the design of future studies.MethodsWe used scoping review methods to examine reproducibility within a cohort of randomized trials examining clinical critical care research and published in the top general medical and critical care journals. To identify relevant clinical practices, we searched the New England Journal of Medicine, The Lancet, and JAMA for randomized trials published up to April 2016. To identify a comprehensive set of studies for these practices, included articles informed secondary searches within other high-impact medical and specialty journals. We included late-phase randomized controlled trials examining therapeutic clinical practices in adults admitted to general medical-surgical or specialty intensive care units (ICUs). Included articles were classified using a reproducibility framework. An original study was the first to evaluate a clinical practice. A reproduction attempt re-evaluated that practice in a new set of participants.ResultsOverall, 158 practices were examined in 275 included articles. A reproduction attempt was identified for 66 practices (42%, 95% CI 33-50%). Original studies reported larger effects than reproduction attempts (primary endpoint, risk difference 16.0%, 95% CI 11.6-20.5% vs. 8.4%, 95% CI 6.0-10.8%, P = 0.003). More than half of clinical practices with a reproduction attempt demonstrated effects that were inconsistent with the original study (56%, 95% CI 42-68%), among which a large number were reported to be efficacious in the original study and to lack efficacy in the reproduction attempt (34%, 95% CI 19-52%). Two practices reported to be efficacious in the original study were found to be harmful in the reproduction attempt.ConclusionsA minority of critical care practices with research published in high-profile journals were evaluated for reproducibility; less than half had reproducible effects.
| S-EPMC5820784 | biostudies-literature
Project description:Amidst a worldwide vaccination campaign, trust in science plays a significant role when addressing the COVID-19 pandemic. Given current concerns regarding research standards, we were interested in how Spanish scholars perceived COVID-19 research and the extent to which questionable research practices and potentially problematic academic incentives are commonplace. We asked researchers to evaluate the expected quality of their COVID-19 projects and other peers' research and compared these assessments with those from scholars not involved in COVID-19 research. We investigated self-admitting and estimated rates of questionable research practices and attitudes towards current research status. Responses from 131 researchers suggested that COVID-19 evaluations followed partisan lines, with scholars being more pessimistic about others' colleagues' research than their own. Additionally,researchers not involved in COVID-19 projects were more negative than their participating peers. These differences were particularly notable for areas such as the expected theoretical foundations or overall quality of the research, among others. Most Spanish scholars expected questionable research practices and inadequate incentives to be widespread. In these two aspects, researchers tended to agree regardless of their involvement in COVID-19 research. We provide specific recommendations for improving future meta-science studies, such as redefining QRPs as inadequate research practices (IRP). This change could help avoid key controversies regarding QRPs' definition while highlighting their detrimental impact. Lastly, we join previous calls to improve transparency and academic career incentives as a cornerstone for generating trust in science.Supplementary informationThe online version contains supplementary material available at 10.1007/s12144-022-02797-6.
| S-EPMC8884412 | biostudies-literature
Project description:Child sexual assault (CSA) cases reliant on uncorroborated testimony yield low conviction rates. Past research demonstrated a strong relationship between verdict and juror CSA knowledge such as typical delays in reporting by victims, and perceived victim credibility. This trial simulation experiment examined the effectiveness of interventions by an expert witness or an educative judicial direction in reducing jurors' CSA misconceptions. Participants were 885 jurors in New South Wales, Australia. After viewing a professionally acted video trial, half the jurors rendered individual verdicts and half deliberated in groups of 8-12 before completing a post-trial questionnaire. Multilevel structural equation modeling exploring the relationship between CSA knowledge and verdict demonstrated that greater CSA knowledge after the interventions increased the odds ratio to convict by itself, and that the judicial direction predicted a higher level of post-trial CSA knowledge in jurors than other expert interventions. Moreover, greater CSA knowledge was associated with heightened credibility perceptions of the complainant and a corroborating witness. At the conclusion of the trial, the more jurors knew about CSA, the higher the perceived credibility of both the complainant and her grandmother, and the more likely jurors were to convict the accused.
| S-EPMC8417353 | biostudies-literature
Project description:Peripheral inflammatory markers (PIMs), such as C-reactive protein (CRP) or white blood cell count (WBC), have been associated with depression severity in meta-analyses and large cohort studies. However, in typically-sized psychoimmunology studies (N < 200) that explore associations between PIMs and neurobiological/psychosocial constructs related to depression and studies that examine less-studied PIMs (e.g., interferon gamma), significant concerns about reproducibility of results exist. For the well-characterized association between PIMs (CRP/WBC) and depression severity, we examined statistical errors as a function of sample size in a large community cohort (n = 24,550). We further assessed how statistical errors varied as related to analytic decisions (e.g., number of covariates) and characteristics related to study design (e.g., relationships within subgroups of patients). Only large samples (e.g., n = 1000 to n = 10,000) were sufficiently powered to detect PIM-depression associations and minimized overestimation of effect sizes (e.g., effect size inflation), and greater sample sizes were required as more covariates were included in analytic models. Moderately sized samples (n > 500) generally ensured the correct directionality of effect sizes (e.g., low rates of sign reversal). Sample sizes required for 80% power also varied widely depending on study design characteristics (e.g., N = 350 to N = 10,000+). Typically-sized psychoimmunology studies examining PIM-depression associations (N < 200) are likely underpowered and at high risk of overestimation of effect sizes. Study design characteristics also notably influence power and statistical error rates. Use of large sample sizes (e.g., N > 7000) and consideration of analytic decisions (e.g., number/choice of covariates) will maximize reproducibility of psychoimmunology studies related to depression to enhance development of treatments for depression or to help understand pathophysiological mechanisms of depression.
| S-EPMC11762288 | biostudies-literature
Project description:Genome editing tools have already revolutionized biomedical research and are also expected to have an important impact in the clinic. However, their extensive use in research has revealed much unpredictability, both off and on target, in the outcome of their application. We discuss the challenges associated with this unpredictability, both for research and in the clinic. For the former, an extensive validation of the model is essential. For the latter, potential unpredicted activity does not preclude the use of these tools but requires that molecular evidence to underpin the relevant risk:benefit evaluation is available. Safe and successful clinical application will also depend on the mode of delivery and the cellular context.
| S-EPMC7264426 | biostudies-literature
Project description:Many researchers try to understand a biological condition by identifying biomarkers. This is typically done using univariate hypothesis testing over a labeled dataset, declaring a feature to be a biomarker if there is a significant statistical difference between its values for the subjects with different outcomes. However, such sets of proposed biomarkers are often not reproducible - subsequent studies often fail to identify the same sets. Indeed, there is often only a very small overlap between the biomarkers proposed in pairs of related studies that explore the same phenotypes over the same distribution of subjects. This paper first defines the Reproducibility Score for a labeled dataset as a measure (taking values between 0 and 1) of the reproducibility of the results produced by a specified fixed biomarker discovery process for a given distribution of subjects. We then provide ways to reliably estimate this score by defining algorithms that produce an over-bound and an under-bound for this score for a given dataset and biomarker discovery process, for the case of univariate hypothesis testing on dichotomous groups. We confirm that these approximations are meaningful by providing empirical results on a large number of datasets and show that these predictions match known reproducibility results. To encourage others to apply this technique to analyze their biomarker sets, we have also created a publicly available website, https://biomarker.shinyapps.io/BiomarkerReprod/, that produces these Reproducibility Score approximations for any given dataset (with continuous or discrete features and binary class labels).
| S-EPMC9333302 | biostudies-literature
Project description:BackgroundReproducibility is the hallmark of good science. Maintaining a high degree of transparency in scientific reporting is essential not just for gaining trust and credibility within the scientific community but also for facilitating the development of new ideas. Sharing data and computer code associated with publications is becoming increasingly common, motivated partly in response to data deposition requirements from journals and mandates from funders. Despite this increase in transparency, it is still difficult to reproduce or build upon the findings of most scientific publications without access to a more complete workflow.FindingsVersion control systems (VCS), which have long been used to maintain code repositories in the software industry, are now finding new applications in science. One such open source VCS, Git, provides a lightweight yet robust framework that is ideal for managing the full suite of research outputs such as datasets, statistical code, figures, lab notes, and manuscripts. For individual researchers, Git provides a powerful way to track and compare versions, retrace errors, explore new approaches in a structured manner, while maintaining a full audit trail. For larger collaborative efforts, Git and Git hosting services make it possible for everyone to work asynchronously and merge their contributions at any time, all the while maintaining a complete authorship trail. In this paper I provide an overview of Git along with use-cases that highlight how this tool can be leveraged to make science more reproducible and transparent, foster new collaborations, and support novel uses.
| S-EPMC3639880 | biostudies-literature