Quality of reporting of randomised controlled trials of artificial intelligence in healthcare: a systematic review
Ontology highlight
ABSTRACT: Objectives
The aim of this study was to evaluate the quality of reporting of randomised controlled trials (RCTs) of artificial intelligence (AI) in healthcare against Consolidated Standards of Reporting Trials—AI (CONSORT-AI) guidelines. Design
Systematic review. Data sources
We searched PubMed and EMBASE databases for studies reported from January 2015 to December 2021. Eligibility criteria
We included RCTs reported in English that used AI as the intervention. Protocols, conference abstracts, studies on robotics and studies related to medical education were excluded. Data extraction
The included studies were graded using the CONSORT-AI checklist, comprising 43 items, by two independent graders. The results were tabulated and descriptive statistics were reported. Results
We screened 1501 potential abstracts, of which 112 full-text articles were reviewed for eligibility. A total of 42 studies were included. The number of participants ranged from 22 to 2352. Only two items of the CONSORT-AI items were fully reported in all studies. Five items were not applicable in more than 85% of the studies. Nineteen per cent (8/42) of the studies did not report more than 50% (21/43) of the CONSORT-AI checklist items. Conclusions
The quality of reporting of RCTs in AI is suboptimal. As reporting is variable in existing RCTs, caution should be exercised in interpreting the findings of some studies.
SUBMITTER: Shahzad R
PROVIDER: S-EPMC9445816 | biostudies-literature | 2022 Jan
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Randomised controlled trials (RCTs) represent a gold standard for evaluating therapeutic interventions. However, poor reporting clarity can prevent readers from assessing potential bias that can arise from a lack of methodological rigour. The Consolidated Standards of Reporting Trials statement for non-pharmacological interventions 2008 (CONSORT NPT) was developed to aid reporting. RCTs in ophthalmic surgery pose particular challenges in study design and implementation. We aim to provide the first assessment of the compliance of RCTs in ophthalmic surgery to the CONSORT NPT statement.In August 2012, the Medline database was searched for RCTs in ophthalmic surgery reported between 1 January 2011 and 31 December 2011. Results were searched by two authors and relevant papers selected. Papers were scored against the 23-item CONSORT NPT checklist and compared against surrogate markers of paper quality. The CONSORT score was also compared between different RCT designs.In all, 186 papers were retrieved. Sixty-five RCTs, involving 5803 patients, met the inclusion criteria. The mean CONSORT score was 8.9 out of 23 (39%, range 3.0-14.7, SD 2.49). The least reported items related to the title and abstract (1.6%), reporting intervention adherence (3.1%), and interpretation of results (4.7%). No significant correlation was found between CONSORT score and journal impact factor (R=0.14, P=0.29), number of authors (R=0.01, P=0.93), or whether the RCT used paired-eye, one-eye, or two-eye designs in their randomisation (P=0.97).The reporting of RCTs in ophthalmic surgery is suboptimal. Further work is needed by trial groups, funding agencies, authors, and journals to improve reporting clarity.
| S-EPMC4274293 | biostudies-other
Project description:Interventions from randomised controlled trials can only be replicated if they are reported in sufficient detail. The results of trials can only be confidently interpreted if the delivery of the intervention was systematic and the protocol adhered to. We systematically reviewed trials of anaesthetic interventions published in 12 journals from January 2016 to September 2019. We assessed the detail with which interventions were reported, using the Consolidated Standards of Reporting Trials statement for non-pharmacological treatments. We analysed 162 interventions reported by 78 trials in 18,675 participants. Detail sufficiently precise to replicate the intervention was reported for 111 (69%) interventions. Intervention standardisation was reported for 135 (83%) out of the 162 interventions, and protocol adherence was reported for 20 (12%) interventions. Sixty (77%) out of the 78 trials reported the administrative context in which interventions were delivered and 36 (46%) trials detailed the expertise of the practitioners. We conclude that bespoke reporting tools should be developed for anaesthetic interventions and interventions in other areas such as critical care.
| S-EPMC8246731 | biostudies-literature
Project description:The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77-94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.
| S-EPMC10883966 | biostudies-literature
Project description:PurposeRandomised controlled trials (RCTs) are considered the gold standard for evaluating the efficacy of an intervention. However, previous research has shown that RCTs in several surgical specialities are poorly reported, making it difficult to ascertain if various biases have been appropriately minimised. This systematic review assesses the reporting quality of surgical head and neck cancer RCTs.MethodsA literature search of PubMed and Embase was performed. Papers were included if they reported RCTs which assessed a surgical technique used to treat or diagnose head and neck cancer published during or after 2011. The CONSORT 2010 checklist was used to evaluate the reporting quality of these trials.Results41 papers were included. The mean CONSORT score was 16.5/25 (66% adherence) and the scores ranged from 7.5 (30%) to 25. The most common omissions were full trial protocol (found in 14.6%), participant recruitment method (22%) and effect size with a precision estimate for all outcome measures (29.3%). The full design and implementation of the randomisation methods were reported in 6 (14.6%). Papers published in journals which endorsed CONSORT had significantly higher scores (p = 0.02) and the journal impact factor was significantly correlated with CONSORT score (p = 0.01).ConclusionWe have identified several pieces of information that are underreported in surgical head and neck cancer RCTs. These omissions make understanding and comparing the methodologies and conclusions of RCTs more difficult. The endorsement of CONSORT by journals improved adherence, suggesting that wider adoption of the checklist may improve reporting.
| S-EPMC8486722 | biostudies-literature
Project description:To determine the quality of randomized controlled clinical trial (RCT) reports in diabetes research in Iran.Systematized review.We included RCTs conducted on diabetes mellitus in Iran. Animal studies, educational interventions, and non-randomized trials were excluded. We excluded duplicated publications reporting the same groups of participants and intervention. Two independent reviewers identify all eligible articles specifically designed data extraction form. We searched through international databases; Scopus, ProQuest, EBSCO, Science Direct, Web of Science, Cochrane Library, PubMed; and national databases (In Persian language) such as Magiran, Scientific Information Database (SID) and IranMedex from January 1995 to January of 2013 Two investigators assessed the quality of reporting by CONSORT 2010 (Consolidated Standards of Reporting Trials) checklist statemen.t,. Discrepancies were resolved by third reviewer consulting.One hundred and eight five (185) studies were included and appraised. Half of them (55.7 %) were published in Iranian journals. Most (89.7 %) were parallel RCTs, and being performed on type2 diabetic patients (77.8 %). Less than half of the CONSORT items (43.2 %) were reported in studies, totally. The reporting of randomization and blinding were poor. A few studies 15.1 % mentioned the method of random sequence generation and strategy of allocation concealment. And only 34.8 % of trials report how blinding was applied.The findings of this study show that the quality of RCTs conducted in Iran in diabetes research seems suboptimal and the reporting is also incomplete however an increasing trend of improvement can be seen over time. Therefore, it is suggested Iranian researchers pay much more attention to design and methodological quality in conducting and reporting of diabetes RCTs.
| S-EPMC5015251 | biostudies-other
Project description:BackgroundThe number of artificial intelligence (AI) studies in medicine has exponentially increased recently. However, there is no clear quantification of the clinical benefits of implementing AI-assisted tools in patient care.ObjectiveThis study aims to systematically review all published randomized controlled trials (RCTs) of AI-assisted tools to characterize their performance in clinical practice.MethodsCINAHL, Cochrane Central, Embase, MEDLINE, and PubMed were searched to identify relevant RCTs published up to July 2021 and comparing the performance of AI-assisted tools with conventional clinical management without AI assistance. We evaluated the primary end points of each study to determine their clinical relevance. This systematic review was conducted following the updated PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines.ResultsAmong the 11,839 articles retrieved, only 39 (0.33%) RCTs were included. These RCTs were conducted in an approximately equal distribution from North America, Europe, and Asia. AI-assisted tools were implemented in 13 different clinical specialties. Most RCTs were published in the field of gastroenterology, with 15 studies on AI-assisted endoscopy. Most RCTs studied biosignal-based AI-assisted tools, and a minority of RCTs studied AI-assisted tools drawn from clinical data. In 77% (30/39) of the RCTs, AI-assisted interventions outperformed usual clinical care, and clinically relevant outcomes improved with AI-assisted intervention in 70% (21/30) of the studies. Small sample size and single-center design limited the generalizability of these studies.ConclusionsThere is growing evidence supporting the implementation of AI-assisted tools in daily clinical practice; however, the number of available RCTs is limited and heterogeneous. More RCTs of AI-assisted tools integrated into clinical practice are needed to advance the role of AI in medicine.Trial registrationPROSPERO CRD42021286539; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=286539.
| S-EPMC9459941 | biostudies-literature
Project description:ObjectiveEvaluate the quality of exercise randomised controlled trial (RCT) reporting and conduct in clinical populations (ie, adults with or at risk of chronic conditions) and compare with matched pharmacological RCTs.DesignSystematic review.Data sourcesEmbase (Elsevier), PubMed (NLM) and CINAHL (EBSCO).Study selectionRCTs of exercise in clinical populations with matching pharmacological RCTs published in leading clinical, medical and specialist journals with impact factors ≥15.Review methodsOverall RCT quality was evaluated by two independent reviewers using three research reporting guidelines (ie, Consolidated Standards of Reporting Trials (CONSORT; pharmacological RCTs)/CONSORT for non-pharmacological treatments; exercise RCTs), CONSORT-Harms, Template for Intervention Description and Replication) and two risk of bias assessment (research conduct) tools (ie, Cochrane Risk of Bias, Jadad Scale). We compared research reporting and conduct quality within exercise RCTs with matched pharmacological RCTs, and examined factors associated with quality in exercise and pharmacological RCTs, separately.FindingsForty-eight exercise RCTs (11 658 patients; median sample n=138) and 48 matched pharmacological RCTs were evaluated (18 501 patients; median sample n=160). RCTs were conducted primarily in cardiovascular medicine (43%) or oncology (31%). Overall quality score (composite of all research reporting and conduct quality scores; primary endpoint) for exercise RCTs was 58% (median score 46 of 80; IQR: 39-51) compared with 77% (53 of 68; IQR: 47-58) in the matched pharmacological RCTs (p≤0.001). Individual quality scores for trial reporting and conduct were lower in exercise RCTs compared with matched pharmacological RCTs (p≤0.03). Factors associated with higher overall quality scores for exercise RCTs were journal impact factor (≥25), sample size (≥152) and publication year (≥2013).Conclusions and relevanceResearch reporting and conduct quality within exercise RCTs is inferior to matched pharmacological RCTs. Suboptimal RCT reporting and conduct impact the fidelity, interpretation, and reproducibility of exercise trials and, ultimately, implementation of exercise in clinical populations.Prospero registration numberCRD42018095033.
| S-EPMC8359527 | biostudies-literature
Project description:ObjectivesTo determine the incidence and characteristics of randomised controlled trials that report using the modified intention to treat approach, and how the approach is described.DesignSystematic review.Data sourcesPubMed, Embase, Cochrane central register of controlled trials, ISI Web of Knowledge, Ovid, HighWire Press, Science-Direct, Ingenta, Medscape, BioMed Central, Springer, and Wiley, from inception to December 2006.Main outcome measuresIncidence of trials in which use of modified intention to treat was reported, and how the approach was described (classified according to the type and number of deviations from the intention to treat approach).Results475 randomised controlled trials reported use of a modified intention to treat analysis. Of these, 76 (16%) were published in five highly cited general medical journals. The incidence of all trials that reported use of modified intention to treat published in journals indexed in Medline increased from 0.006% in 1982-6 to 0.5% in 2002-6 (P<0.001 for linear trend). When the description of the modified intention to treat was examined in each trial, 192 (40%) reported one type of deviation from the intention to treat approach, 261 (55%) reported two or more types, and 22 (5%) did not describe any type. In 266 (56%) of the trials the deviation was related to the treatment received, in 196 (41%) to a post baseline assessment, in 118 (25%) to a baseline assessment, in 108 (23%) to a target condition, and in 23 (5%) to follow-up. Post-randomisation exclusions occurred in 380 (80%) trials. The results reported by 270 of the 352 (77%) superiority trials favoured the drug under investigation. All of the 123 trials using equivalence or non-inferiority methods to investigate interventions reported results that favoured their assumptions.ConclusionsRandomised controlled trials that report using a modified intention to treat are increasingly being published in the medical literature. The descriptions of such an approach were ambiguous, and may cover any type of descriptions for exclusion, such as missing data and deviation from protocol. Explicit statements about post-randomisation exclusions should replace the ambiguous terminology of modified intention to treat.
| S-EPMC2885592 | biostudies-literature
Project description:The evidence of the impact of traditional statistical (TS) and artificial intelligence (AI) tool interventions in clinical practice was limited. This study aimed to investigate the clinical impact and quality of randomized controlled trials (RCTs) involving interventions evaluating TS, machine learning (ML), and deep learning (DL) prediction tools. A systematic review on PubMed was conducted to identify RCTs involving TS/ML/DL tool interventions in the past decade. A total of 65 RCTs from 26,082 records were included. A majority of them had model development studies and generally good performance was achieved. The function of TS and ML tools in the RCTs mainly included assistive treatment decisions, assistive diagnosis, and risk stratification, but DL trials were only conducted for assistive diagnosis. Nearly two-fifths of the trial interventions showed no clinical benefit compared to standard care. Though DL and ML interventions achieved higher rates of positive results than TS in the RCTs, in trials with low risk of bias (17/65) the advantage of DL to TS was reduced while the advantage of ML to TS disappeared. The current applications of DL were not yet fully spread performed in medicine. It is predictable that DL will integrate more complex clinical problems than ML and TS tools in the future. Therefore, rigorous studies are required before the clinical application of these tools.
| S-EPMC8553754 | biostudies-literature
Project description:BackgroundMany randomised controlled trials (RCTs) of herbal interventions have been conducted in the ASEAN Communities. Good quality reporting of RCTs is essential for assessing clinical significance. Given the importance ASEAN placed on herbal medicines, the reporting quality of RCTs of herbal interventions among the ASEAN Communities deserved a special attention.ObjectivesTo systematically review the quality of reporting of RCTs of herbal interventions conducted in the ASEAN Plus Six Countries.MethodsSearches were performed using PubMed, EMBASE, The Cochrane Library, and Allied and Complementary Medicine (AMED), from inception through October 2013. These were limited to studies specific to humans and RCTs. Herbal species search terms were based on those listed in the National List of Essential Medicines [NLEM (Thailand, 2011)]. Studies conducted in the ASEAN Plus Six Countries, published in English were included.ResultsSeventy-one articles were identified. Thirty (42.25%) RCTs were from ASEAN Countries, whereas 41 RCTs (57.75%) were from Plus Six Group. Adherence to the recommended CONSORT checklist items for reporting of RCTs of herbal interventions among ASEAN Plus Six Countries ranged from 0% to 97.18%. Less than a quarter of the RCTs (18.31%) reported information on standardisation of the herbal products. However, the scope of our interventions of interest was limited to those developed from 20 herbal species listed in the NLEM of Thailand.ConclusionsThe present study highlights the need to improve reporting quality of RCTs of herbal interventions across ASEAN Plus Six Communities.
| S-EPMC4310614 | biostudies-literature