Dataset Information

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system.

ABSTRACT:

Background

Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers' needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment.

Methods

We assessed the feasibility of adopting RobotReviewer within a national public health institute using a randomized, real-time, user-centered study. The study included 26 RCTs and six reviewers from two projects examining health and social interventions. We randomized these studies to one of two RobotReviewer platforms. We operationalized feasibility as accuracy, time use, and reviewer acceptability. We measured accuracy by the number of corrections made by human reviewers (either to automated assessments or another human reviewer's assessments). We explored acceptability through group discussions and individual email responses after presenting the quantitative results.

Results

Reviewers were equally likely to accept judgment by RobotReviewer as each other's judgement during the consensus process when measured dichotomously; risk ratio 1.02 (95% CI 0.92 to 1.13; p = 0.33). We were not able to compare time use. The acceptability of the program by researchers was mixed. Less experienced reviewers were generally more positive, and they saw more benefits and were able to use the tool more flexibly. Reviewers positioned human input and human-to-human interaction as superior to even a semi-automation of this process.

Conclusion

Despite being presented with evidence of RobotReviewer's equal performance to humans, participating reviewers were not interested in modifying standard procedures to include automation. If further studies confirm equal accuracy and reduced time compared to manual practices, we suggest that the benefits of RobotReviewer may support its future implementation as one of two assessors, despite reviewer ambivalence. Future research should study barriers to adopting automated tools and how highly educated and experienced researchers can adapt to a job market that is increasingly challenged by new technologies.

SUBMITTER: Jardim PSJ

PROVIDER: S-EPMC9174024 | biostudies-literature | 2022 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system.

Jardim Patricia Sofia Jacobsen PSJ Rose Christopher James CJ Ames Heather Melanie HM Echavez Jose Francisco Meneses JFM Van de Velde Stijn S Muller Ashley Elizabeth AE

BMC medical research methodology 20220608 1

<h4>Background</h4>Machine learning and automation are increasingly used to make the evidence synthesis process faster and more responsive to policymakers' needs. In systematic reviews of randomized controlled trials (RCTs), risk of bias assessment is a resource-intensive task that typically requires two trained reviewers. One function of RobotReviewer, an off-the-shelf machine learning system, is an automated risk of bias assessment.<h4>Methods</h4>We assessed the feasibility of adopting RobotR ...[more]

PMID: 35676632

Similar Datasets

Project description:BackgroundThere is increasing evidence that research misbehaviour is common, especially the minor forms. Previous studies on research misbehaviour primarily focused on biomedical and social sciences, and evidence from natural sciences and humanities is scarce. We investigated what academic researchers in Amsterdam perceived to be detrimental research misbehaviours in their respective disciplinary fields.MethodsWe used an explanatory sequential mixed methods design. First, survey participants from four disciplinary fields rated perceived frequency and impact of research misbehaviours from a list of 60. We then combined these into a top five ranking of most detrimental research misbehaviours at the aggregate level, stratified by disciplinary field. Second, in focus group interviews, participants from each academic rank and disciplinary field were asked to reflect on the most relevant research misbehaviours for their disciplinary field. We used participative ranking methodology inducing participants to obtain consensus on which research misbehaviours are most detrimental.ResultsIn total, 1080 researchers completed the survey (response rate: 15%) and 61 participated in the focus groups (3 three to 8 eight researchers per group). Insufficient supervision consistently ranked highest in the survey regardless of disciplinary field and the focus groups confirmed this. Important themes in the focus groups were insufficient supervision, sloppy science, and sloppy peer review. Biomedical researchers and social science researchers were primarily concerned with sloppy science and insufficient supervision. Natural sciences and humanities researchers discussed sloppy reviewing and theft of ideas by reviewers, a form of plagiarism. Focus group participants further provided examples of particular research misbehaviours they were confronted with and how these impacted their work as a researcher.ConclusionWe found insufficient supervision and various forms of sloppy science to score highly on aggregate detrimental impact throughout all disciplinary fields. Researchers from the natural sciences and humanities also perceived nepotism to be of major impact on the aggregate level. The natural sciences regarded fabrication of data of major impact as well. The focus group interviews helped to understand how researchers interpreted 'insufficient supervision'. Besides, the focus group participants added insight into sloppy science in practice. Researchers from the natural sciences and humanities added new research misbehaviours concerning their disciplinary fields to the list, such as the stealing of ideas before publication. This improves our understanding of research misbehaviour beyond the social and biomedical fields.

Project description:BackgroundSystematic reviews (SRs) are cornerstones of evidence-based medicine and have contributed significantly to breakthroughs since the 1980's. However, preclinical SRs remain relatively rare despite their many advantages. Since 2011 the Dutch health funding organisation (ZonMw) has run a grant scheme dedicated to promoting the training, coaching and conduct of preclinical SRs. Our study focuses on this funding scheme to investigate the relevance, effects and benefits of conducting preclinical SRs on researchers and their research.MethodsWe recruited researchers who attended funded preclinical SR workshops and who conducted, are still conducting, or prematurely stopped a SR with funded coaching. We gathered data using online questionnaires followed by semi-structured interviews. Both aimed to explore the impact of conducting a SR on researchers' subsequent work, attitudes, and views about their research field. Data-analysis was performed using Excel and ATLAS.ti.ResultsConducting preclinical SRs had two distinct types of impact. First, the researchers acquired new skills and insights, leading to a change in mindset regarding the quality of animal research. This was mainly seen in the way participants planned, conducted and reported their subsequent animal studies, which were more transparent and of a higher quality than their previous work. Second, participants were eager to share their newly acquired knowledge within their laboratories and to advocate for change within their research teams and fields of interest. In particular, they emphasised the need for preclinical SRs and improved experimental design within preclinical research, promoting these through education and published opinion papers.ConclusionBeing trained and coached in the conduct of preclinical SRs appears to be a contributing factor to many beneficial changes which will impact the quality of preclinical research in the long-term. Our findings suggest that this ZonMw funding scheme is helpful in improving the quality and transparency of preclinical research. Similar funding schemes should be encouraged, preferably by a broader group of funders or financers, in the future.

Project description:IntroductionReal-world evidence is important in regulatory and funding decisions. Manual data extraction from electronic health records (EHRs) is time-consuming and challenging to maintain. Automated extraction using natural language processing (NLP) and artificial intelligence may facilitate this process. Whereas NLP offers a faster solution than manual methods of extraction, the validity of extracted data remains in question. The current study compared manual and automated data extraction from the EHR of patients with advanced lung cancer.MethodsPreviously, we extracted EHRs from 1209 patients diagnosed with advanced lung cancer (stage IIIB or IV) between January 2015 and December 2017 at Princess Margaret Cancer Centre (Toronto, Canada) using the commercially available artificial intelligence engine, DARWEN (Pentavere, Ontario, Canada). For comparison, 100 of 333 patients that received systemic therapy were randomly selected and clinical data manually extracted by two trained abstractors using the same accepted gold standard feature definitions, including patient, disease characteristics, and treatment data. All cases were re-reviewed by an expert adjudicator. Accuracy and concordance between automated and manual methods are reported.ResultsAutomated extraction required considerably less time (<1 day) than manual extraction (∼225 person-hr). The collection of demographic data (age, sex, diagnosis) was highly accurate and concordant with both methods (96%-100%). Accuracy (for either extraction approach) and concordance were lower for unstructured data elements in EHR, such as performance status, date of diagnosis, and smoking status (NLP accuracy: 88%-94%; Manual accuracy: 78%-94%; concordance: 71%-82%). Concurrent medications (86%-100%) and comorbid conditions (96%-100%), were reported with high accuracy and concordance. Treatment details were also accurately captured with both methods (84%-100%) and highly concordant (83%-99%). Detection of whether biomarker testing was performed was highly accurate and concordant (96%-98%), although detection of biomarker test results was more variable (accuracy 84%-100%, concordance 84%-99%). Features with syntactic or semantic variation requiring clinical interpretation were extracted with slightly lower accuracy by both NLP and manual review. For example, metastatic sites were more accurately identified through NLP extraction (NLP: 88%-99%; manual: 71%-100%; concordance: 70%-99%) with the exception of lung and lymph node metastases (NLP: 66%-71%; manual: 87%-92%; concordance: 58%) owing to analogous terms used in radiology reports not being included in the accepted gold standard definition.ConclusionsAutomated data abstraction from EHR is highly accurate and faster than manual abstraction. Key challenges include poorly structured EHR and the use of analogous terms beyond the accepted gold standard definition. The application of NLP can facilitate real-world evidence studies at a greater scale than could be achieved with manual data extraction.

Project description:BackgroundOlder adults now make up about two-thirds of hospital admissions, with up to 50% experiencing cognitive impairments such as dementia. These patients often struggle with adherence to care plans and maintaining regular day or night cycles, presenting challenges for nurses. Hospitals are typically unprepared to manage this patient population, resulting in increased nurse workload and challenges like managing motor agitation, which can lead to falls or accidental removal of medical devices.ObjectiveThis study aimed to (1) assess how an in-bed real-time motion monitoring system (IRMS) impacts nurses' perceptions of physical and mental stress, (2) evaluate the IRMS's effect on the care process, (3) explore ethical implications like patient autonomy and privacy, and (4) understand how nurses acquire knowledge about the technology and how this affects their assessment of the IRMS.MethodsThe IRMS, which provides real-time motion monitoring and bed edge or exit information, was implemented in the geriatric ward of a university medical center. The study followed a monocentric, explorative evaluation design using a mixed methods approach. It lasted 24 weeks and had two phases. In Phase 0 (6 weeks), patients received standard care. In Phase 1 (18 weeks), the IRMS was introduced. Initial data were gathered through focus groups and participant observations during manufacturer training sessions. At the end of the intervention, a survey, a second focus group, and an interview were conducted to capture nurses' experiences. The study follows the Good Reporting of a Mixed Method Study (GRAMMS) checklist for reporting.ResultsInitial training sessions with 12 participants (10 nurses and 2 physiotherapists) showed varying levels of engagement, with the second session demonstrating more optimism and interprofessional collaboration. A total of 10 questionnaires were completed (10/21, 48%). Survey results showed that 80% (8/10) of nurses found the IRMS valuable for assessing the quality of work, and 90% (9/10) were willing to continue using it. The system was regarded as reliable for monitoring bed edge and exit events. Usability was positively rated, with minimal concerns about documentation burden. Focus group discussions (n=3 per session) indicated that nurses viewed the system as reliable and appreciated its role in reducing anxiety related to fall prevention. However, concerns about patient privacy and monitoring were raised. Nurses expressed a willingness to continue using the IRMS but reaffirmed their ability to care for patients without it.ConclusionsNurses had a generally positive attitude toward the IRMS, recognizing its benefits, particularly for nighttime monitoring. Although its effectiveness in preventing falls remains inconclusive, the system helps reduce nurses' fear of falls and enhances their responsiveness. The study highlights the broader impact of the IRMS beyond fall prevention and stresses the importance of thoughtful integration into health care practice.

Dataset Information

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system.

Background

Methods

Results

Conclusion

Publications

Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets