Dataset Information

Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge.

ABSTRACT: Polyps are well-known cancer precursors identified by colonoscopy. However, variability in their size, appearance, and location makes the detection of polyps challenging. Moreover, colonoscopy surveillance and removal of polyps are highly operator-dependent procedures and occur in a highly complex organ topology. There exists a high missed detection rate and incomplete removal of colonic polyps. To assist in clinical procedures and reduce missed rates, automated methods for detecting and segmenting polyps using machine learning have been achieved in past years. However, the major drawback in most of these methods is their ability to generalise to out-of-sample unseen datasets from different centres, populations, modalities, and acquisition systems. To test this hypothesis rigorously, we, together with expert gastroenterologists, curated a multi-centre and multi-population dataset acquired from six different colonoscopy systems and challenged the computational expert teams to develop robust automated detection and segmentation methods in a crowd-sourcing Endoscopic computer vision challenge. This work put forward rigorous generalisability tests and assesses the usability of devised deep learning methods in dynamic and actual clinical colonoscopy procedures. We analyse the results of four top performing teams for the detection task and five top performing teams for the segmentation task. Our analyses demonstrate that the top-ranking teams concentrated mainly on accuracy over the real-time performance required for clinical applicability. We further dissect the devised methods and provide an experiment-based hypothesis that reveals the need for improved generalisability to tackle diversity present in multi-centre datasets and routine clinical procedures.

SUBMITTER: Ali S

PROVIDER: S-EPMC10805888 | biostudies-literature | 2024 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge.

Ali Sharib S Ghatwary Noha N Jha Debesh D Isik-Polat Ece E Polat Gorkem G Yang Chen C Li Wuyang W Galdran Adrian A Ballester Miguel-Ángel González MG Thambawita Vajira V Hicks Steven S Poudel Sahadev S Lee Sang-Woong SW Jin Ziyi Z Gan Tianyuan T Yu ChengHui C Yan JiangPeng J Yeo Doyeob D Lee Hyunseok H Tomar Nikhil Kumar NK Haithami Mahmood M Ahmed Amr A Riegler Michael A MA Daul Christian C Halvorsen Pål P Rittscher Jens J Salem Osama E OE Lamarque Dominique D Cannizzaro Renato R Realdon Stefano S de Lange Thomas T East James E JE

Scientific reports 20240123 1

Polyps are well-known cancer precursors identified by colonoscopy. However, variability in their size, appearance, and location makes the detection of polyps challenging. Moreover, colonoscopy surveillance and removal of polyps are highly operator-dependent procedures and occur in a highly complex organ topology. There exists a high missed detection rate and incomplete removal of colonic polyps. To assist in clinical procedures and reduce missed rates, automated methods for detecting and segment ...[more]

PMID: 38263232

Similar Datasets

Project description:BackgroundArtificial intelligence in colonoscopy is an emerging field, and its application may help colonoscopists improve inspection quality and reduce the rate of missed polyps and adenomas. Several deep learning-based computer-assisted detection (CADe) techniques were established from small single-center datasets, and unrepresentative learning materials might confine their application and generalization in wide practice. Although CADes have been reported to identify polyps in colonoscopic images and videos in real time, their diagnostic performance deserves to be further validated in clinical practice.AimTo train and test a CADe based on multicenter high-quality images of polyps and preliminarily validate it in clinical colonoscopies.MethodsWith high-quality screening and labeling from 55 qualified colonoscopists, a dataset consisting of over 71000 images from 20 centers was used to train and test a deep learning-based CADe. In addition, the real-time diagnostic performance of CADe was tested frame by frame in 47 unaltered full-ranged videos that contained 86 histologically confirmed polyps. Finally, we conducted a self-controlled observational study to validate the diagnostic performance of CADe in real-world colonoscopy with the main outcome measure of polyps per colonoscopy in Changhai Hospital.ResultsThe CADe was able to identify polyps in the test dataset with 95.0% sensitivity and 99.1% specificity. For colonoscopy videos, all 86 polyps were detected with 92.2% sensitivity and 93.6% specificity in frame-by-frame analysis. In the prospective validation, the sensitivity of CAD in identifying polyps was 98.4% (185/188). Folds, reflections of light and fecal fluid were the main causes of false positives in both the test dataset and clinical colonoscopies. Colonoscopists can detect more polyps (0.90 vs 0.82, P < 0.001) and adenomas (0.32 vs 0.30, P = 0.045) with the aid of CADe, particularly polyps < 5 mm and flat polyps (0.65 vs 0.57, P < 0.001; 0.74 vs 0.67, P = 0.001, respectively). However, high efficacy is not realized in colonoscopies with inadequate bowel preparation and withdrawal time (P = 0.32; P = 0.16, respectively).ConclusionCADe is feasible in the clinical setting and might help endoscopists detect more polyps and adenomas, and further confirmation is warranted.

Project description:Deep learning has shown significant value in automating radiological diagnostics but can be limited by a lack of generalizability to external datasets. Leveraging the geometric principles of non-Euclidean space, certain geometric deep learning approaches may offer an alternative means of improving model generalizability. This study investigates the potential advantages of hyperbolic convolutional neural networks (HCNNs) over traditional convolutional neural networks (CNNs) in neuroimaging tasks. We conducted a comparative analysis of HCNNs and CNNs across various medical imaging modalities and diseases, with a focus on a compiled multi-modality neuroimaging dataset. The models were assessed for their performance parity, robustness to adversarial attacks, semantic organization of embedding spaces, and generalizability. Zero-shot evaluations were also performed with ischemic stroke non-contrast CT images. HCNNs matched CNNs' performance in less complex settings and demonstrated superior semantic organization and robustness to adversarial attacks. While HCNNs equaled CNNs in out-of-sample datasets identifying Alzheimer's disease, in zero-shot evaluations, HCNNs outperformed CNNs and radiologists. HCNNs deliver enhanced robustness and organization in neuroimaging data. This likely underlies why, while HCNNs perform similarly to CNNs with respect to in-sample tasks, they confer improved generalizability. Nevertheless, HCNNs encounter efficiency and performance challenges with larger, complex datasets. These limitations underline the need for further optimization of HCNN architectures. HCNNs present promising improvements in generalizability and resilience for medical imaging applications, particularly in neuroimaging. Despite facing challenges with larger datasets, HCNNs enhance performance under adversarial conditions and offer better semantic organization, suggesting valuable potential in generalizable deep learning models in medical imaging and neuroimaging diagnostics.

Project description:ObjectivesThe purpose of this study was to build a deep learning model to derive labels from neuroradiology reports and assign these to the corresponding examinations, overcoming a bottleneck to computer vision model development.MethodsReference-standard labels were generated by a team of neuroradiologists for model training and evaluation. Three thousand examinations were labelled for the presence or absence of any abnormality by manually scrutinising the corresponding radiology reports ('reference-standard report labels'); a subset of these examinations (n = 250) were assigned 'reference-standard image labels' by interrogating the actual images. Separately, 2000 reports were labelled for the presence or absence of 7 specialised categories of abnormality (acute stroke, mass, atrophy, vascular abnormality, small vessel disease, white matter inflammation, encephalomalacia), with a subset of these examinations (n = 700) also assigned reference-standard image labels. A deep learning model was trained using labelled reports and validated in two ways: comparing predicted labels to (i) reference-standard report labels and (ii) reference-standard image labels. The area under the receiver operating characteristic curve (AUC-ROC) was used to quantify model performance. Accuracy, sensitivity, specificity, and F1 score were also calculated.ResultsAccurate classification (AUC-ROC > 0.95) was achieved for all categories when tested against reference-standard report labels. A drop in performance (ΔAUC-ROC > 0.02) was seen for three categories (atrophy, encephalomalacia, vascular) when tested against reference-standard image labels, highlighting discrepancies in the original reports. Once trained, the model assigned labels to 121,556 examinations in under 30 min.ConclusionsOur model accurately classifies head MRI examinations, enabling automated dataset labelling for downstream computer vision applications.Key points• Deep learning is poised to revolutionise image recognition tasks in radiology; however, a barrier to clinical adoption is the difficulty of obtaining large labelled datasets for model training. • We demonstrate a deep learning model which can derive labels from neuroradiology reports and assign these to the corresponding examinations at scale, facilitating the development of downstream computer vision models. • We rigorously tested our model by comparing labels predicted on the basis of neuroradiology reports with two sets of reference-standard labels: (1) labels derived by manually scrutinising each radiology report and (2) labels derived by interrogating the actual images.

Dataset Information

Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge.

Publications

Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets