Dataset Information

Inter-rater reliability for assessing intracranial collaterals in patients with acute ischemic stroke: comparing 29 raters and an artificial intelligence-based software.

ABSTRACT:

Purpose

Outcome of endovascular treatment in acute ischemic stroke patients is depending on the collateral circulation maintaining blood flow to the ischemic territory. We evaluated the inter-rater reliability and accuracy of raters and an automated algorithm for assessing the collateral score (CS, range: 0-3) in acute ischemic stroke patients.

Methods

Baseline CTA scans with an intracranial anterior occlusion from the MR CLEAN study (n=500) were used. For each core lab CS, ten CTA scans with sufficient quality were randomly selected. After a training session in collateral scoring, all selected CTA scans were individually evaluated for a visual CS by three groups: 7 radiologists, 13 junior and 9 senior radiology residents. Two additional radiologists scored CS to be used as reference, with a third providing a CS to produce a 2 out of 3 consensus CS in case of disagreement. An automated algorithm was also used to compute CS. Inter-rater agreement was reported with intraclass correlation coefficient (ICC). Accuracy of visual and automated CS were calculated.

Results

39 CTA scans were assessed (1 corrupt CTA-scan excluded). All groups showed a moderate ICC (0.689-0.780) in comparison to the reference standard. Overall human accuracy was 65± 7% and increased to 88± 5% for dichotomized CS (0-1, 2-3). Automated CS accuracy was 62%, and 90% for dichotomized CS. No significant difference in accuracy was found between groups with different levels of expertise.

Conclusion

After training, inter-rater reliability in collateral scoring was not influenced by experience. Automated CS performs similar to residents and radiologists in determining a collateral score.

SUBMITTER: Wolff L

PROVIDER: S-EPMC9643213 | biostudies-literature | 2022 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Inter-rater reliability for assessing intracranial collaterals in patients with acute ischemic stroke: comparing 29 raters and an artificial intelligence-based software.

Wolff Lennard L Su Jiahang J Van Loon Derek D van Es Adriaan A van Doormaal Pieter Jan PJ Majoie Charles C van Zwam Wim W Dippel Diederik D van der Lugt Aad A van Walsum Theo T

Neuroradiology 20220524 12

<h4>Purpose</h4>Outcome of endovascular treatment in acute ischemic stroke patients is depending on the collateral circulation maintaining blood flow to the ischemic territory. We evaluated the inter-rater reliability and accuracy of raters and an automated algorithm for assessing the collateral score (CS, range: 0-3) in acute ischemic stroke patients.<h4>Methods</h4>Baseline CTA scans with an intracranial anterior occlusion from the MR CLEAN study (n=500) were used. For each core lab CS, ten CT ...[more]

PMID: 35608629

Similar Datasets

Project description:BackgroundFor diagnostic procedures to be clinically useful, they must be reliable. The interpretation of lumbar spine MRI scans is subject to variability and there is a lack of studies where reliability of multiple degenerative pathologies are rated simultaneously. The objective of our study was to determine the inter-rater reliability of three independent raters evaluating degenerative pathologies seen with lumbar spine MRI.MethodsFifty-nine people, 35 patients with low back pain (LBP) or LBP and leg pain and 24 people without LBP or leg pain, received an MRI of the lumbar spine. Three raters (one radiologist and two chiropractors) evaluated the MRIs for the presence and severity of eight degenerative spinal pathologies using a standardized format: Spondylolisthesis, scoliosis, annular fissure, disc degeneration, disc contour, nerve root compromise, spinal stenosis and facet joint degeneration. Findings were identified and classified at disc level according to type and severity. Raters were instructed to evaluate all study sample persons once to assess inter-rater reliability (fully crossed design). Reliability was calculated using Gwet's Agreement Coefficients (AC1 and AC2) and Cohen's Kappa (κ) and Conger's extension of Cohen's. Gwet's probabilistic benchmarking method to the Landis and Koch scale was used. MRI-findings achieving substantial reliability was considered acceptable.ResultsInter-rater reliability for all raters combined, ranged from (Gwet's AC1 or AC2): 0.64-0.99 and according to probabilistic benchmarking to the Landis and Koch scale equivalent to moderate to almost perfect reliability. Overall reliability level for individual pathologies was almost perfect reliability for spondylolisthesis, spinal stenosis, scoliosis and annular fissure, substantial for nerve root compromise and disc degeneration, and moderate for facet joint degeneration and disc contour.ConclusionInter-rater reliability for 3 raters, evaluating 177 disc levels, was found to be overall acceptable for 6 out of 8 degenerative MRI-findings in the lumbar spine. Ratings of facet joint degeneration and disc contour achieved moderate reliability and was considered unacceptable.

Project description:BACKGROUND:Although previous research has highlighted the association between the built environment and individual health, methodological challenges in assessing the built environment remain. In particular, many researchers have demonstrated the high inter-rater reliability of assessing large or objective built environment features and the low inter-rater reliability of assessing small or subjective built environment features using Google Street View. New methods for auditing the built environment must be evaluated to understand if there are alternative tools through which researchers can assess all types of built environment features with high agreement. This paper investigates measures of inter-rater reliability of GigaPan®, a tool that assists with capturing high-definition panoramic images, relative to Google Street View. METHODS:Street segments (n?=?614) in Pittsburgh, Pennsylvania in the United States were randomly selected to audit using GigaPan® and Google Street View. Each audit assessed features related to land use, traffic and safety, and public amenities. Inter-rater reliability statistics, including percent agreement, Cohen's kappa, and the prevalence-adjusted bias-adjusted kappa (PABAK) were calculated for 106 street segments that were coded by two, different, human auditors. RESULTS:Most large-scale, objective features (e.g. bus stop presence or stop sign presence) demonstrated at least substantial inter-rater reliability for both methods, but significant differences emerged across finely detailed features (e.g. trash) and features at segment endpoints (e.g. sidewalk continuity). After adjusting for the effects of bias and prevalence, the inter-rater reliability estimates were consistently higher for almost all built environment features across GigaPan® and Google Street View. CONCLUSION:GigaPan® is a reliable, alternative audit tool to Google Street View for studying the built environment. GigaPan® may be particularly well-suited for built environment projects with study settings in areas where Google Street View imagery is nonexistent or updated infrequently. The potential for enhanced, detailed imagery using GigaPan® will be most beneficial in studies in which current, time sensitive data are needed or microscale built environment features would be challenging to see in Google Street View. Furthermore, to better understand the effects of prevalence and bias in future reliability studies, researchers should consider using PABAK to supplement or expand upon Cohen's kappa findings.

Dataset Information

Inter-rater reliability for assessing intracranial collaterals in patients with acute ischemic stroke: comparing 29 raters and an artificial intelligence-based software.

Purpose

Methods

Results

Conclusion

Publications

Inter-rater reliability for assessing intracranial collaterals in patients with acute ischemic stroke: comparing 29 raters and an artificial intelligence-based software.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets