Dataset Information

A Competition, Benchmark, Code, and Data for Using Artificial Intelligence to Detect Lesions in Digital Breast Tomosynthesis.

ABSTRACT:

Importance

An accurate and robust artificial intelligence (AI) algorithm for detecting cancer in digital breast tomosynthesis (DBT) could significantly improve detection accuracy and reduce health care costs worldwide.

Objectives

To make training and evaluation data for the development of AI algorithms for DBT analysis available, to develop well-defined benchmarks, and to create publicly available code for existing methods.

Design, setting, and participants

This diagnostic study is based on a multi-institutional international grand challenge in which research teams developed algorithms to detect lesions in DBT. A data set of 22 032 reconstructed DBT volumes was made available to research teams. Phase 1, in which teams were provided 700 scans from the training set, 120 from the validation set, and 180 from the test set, took place from December 2020 to January 2021, and phase 2, in which teams were given the full data set, took place from May to July 2021.

Main outcomes and measures

The overall performance was evaluated by mean sensitivity for biopsied lesions using only DBT volumes with biopsied lesions; ties were broken by including all DBT volumes.

Results

A total of 8 teams participated in the challenge. The team with the highest mean sensitivity for biopsied lesions was the NYU B-Team, with 0.957 (95% CI, 0.924-0.984), and the second-place team, ZeDuS, had a mean sensitivity of 0.926 (95% CI, 0.881-0.964). When the results were aggregated, the mean sensitivity for all submitted algorithms was 0.879; for only those who participated in phase 2, it was 0.926.

Conclusions and relevance

In this diagnostic study, an international competition produced algorithms with high sensitivity for using AI to detect lesions on DBT images. A standardized performance benchmark for the detection task using publicly available clinical imaging data was released, with detailed descriptions and analyses of submitted algorithms accompanied by a public release of their predictions and code for selected methods. These resources will serve as a foundation for future research on computer-assisted diagnosis methods for DBT, significantly lowering the barrier of entry for new researchers.

SUBMITTER: Konz N

PROVIDER: S-EPMC9951043 | biostudies-literature | 2023 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A Competition, Benchmark, Code, and Data for Using Artificial Intelligence to Detect Lesions in Digital Breast Tomosynthesis.

Konz Nicholas N Buda Mateusz M Gu Hanxue H Saha Ashirbani A Yang Jichen J Chledowski Jakub J Park Jungkyu J Witowski Jan J Geras Krzysztof J KJ Shoshan Yoel Y Gilboa-Solomon Flora F Khapun Daniel D Ratner Vadim V Barkan Ella E Ozery-Flato Michal M Martí Robert R Omigbodun Akinyinka A Marasinou Chrysostomos C Nakhaei Noor N Hsu William W Sahu Pranjal P Hossain Md Belayat MB Lee Juhun J Santos Carlos C Przelaskowski Artur A Kalpathy-Cramer Jayashree J Bearce Benjamin B Cha Kenny K Farahani Keyvan K Petrick Nicholas N Hadjiiski Lubomir L Drukker Karen K Armato Samuel G SG Mazurowski Maciej A MA

JAMA network open 20230201 2

<h4>Importance</h4>An accurate and robust artificial intelligence (AI) algorithm for detecting cancer in digital breast tomosynthesis (DBT) could significantly improve detection accuracy and reduce health care costs worldwide.<h4>Objectives</h4>To make training and evaluation data for the development of AI algorithms for DBT analysis available, to develop well-defined benchmarks, and to create publicly available code for existing methods.<h4>Design, setting, and participants</h4>This diagnostic ...[more]

PMID: 36821110

Similar Datasets

Project description:ObjectivesDigital breast tomosynthesis (DBT) can detect more cancers than the current standard breast screening method, digital mammography (DM); however, it can substantially increase the reading workload and thus hinder implementation in screening. Artificial intelligence (AI) might be a solution. The aim of this study was to retrospectively test different ways of using AI in a screening workflow.MethodsAn AI system was used to analyse 14,772 double-read single-view DBT examinations from a screening trial with paired DM double reading. Three scenarios were studied: if AI can identify normal cases that can be excluded from human reading; if AI can replace the second reader; if AI can replace both readers. The number of detected cancers and false positives was compared with DM or DBT double reading.ResultsBy excluding normal cases and only reading 50.5% (7460/14,772) of all examinations, 95% (121/127) of the DBT double reading detected cancers could be detected. Compared to DM screening, 27% (26/95) more cancers could be detected (p < 0.001) while keeping recall rates at the same level. With AI replacing the second reader, 95% (120/127) of the DBT double reading detected cancers could be detected-26% (25/95) more than DM screening (p < 0.001)-while increasing recall rates by 53%. AI alone with DBT has a sensitivity similar to DM double reading (p = 0.689).ConclusionAI can open up possibilities for implementing DBT screening and detecting more cancers with the total reading workload unchanged. Considering the potential legal and psychological implications, replacing the second reader with AI would probably be most the feasible approach.Key points• Breast cancer screening with digital breast tomosynthesis and artificial intelligence can detect more cancers than mammography screening without increasing screen-reading workload. • Artificial intelligence can either exclude low-risk cases from double reading or replace the second reader. • Retrospective study based on paired mammography and digital breast tomosynthesis screening data.

Project description:PurposeTo investigate how an artificial intelligence (AI) system performs at digital mammography (DM) from a screening population with ground truth defined by digital breast tomosynthesis (DBT), and whether AI could detect breast cancers at DM that had originally only been detected at DBT.Materials and methodsIn this secondary analysis of data from a prospective study, DM examinations from 14 768 women (mean age, 57 years), examined with both DM and DBT with independent double reading in the Malmӧ Breast Tomosynthesis Screening Trial (MBTST) (ClinicalTrials.gov: NCT01091545; data collection, 2010-2015), were analyzed with an AI system. Of 136 screening-detected cancers, 95 cancers were detected at DM and 41 cancers were detected only at DBT. The system identifies suspicious areas in the image, scored 1-100, and provides a risk score of 1 to 10 for the whole examination. A cancer was defined as AI detected if the cancer lesion was correctly localized and scored at least 62 (threshold determined by the AI system developers), therefore resulting in the highest examination risk score of 10. Data were analyzed with descriptive statistics, and detection performance was analyzed with receiver operating characteristics.ResultsThe highest examination risk score was assigned to 10% (1493 of 14 786) of the examinations. With 90.8% specificity, the AI system detected 75% (71 of 95) of the DM-detected cancers and 44% (18 of 41) of cancers at DM that had originally been detected only at DBT. The majority were invasive cancers (17 of 18).ConclusionAlmost half of the additional DBT-only screening-detected cancers in the MBTST were detected at DM with AI. AI did not reach double reading performance; however, if combined with double reading, AI has the potential to achieve a substantial portion of the benefit of DBT screening.Keywords: Computer-aided Diagnosis, Mammography, Breast, Diagnosis, Classification, Application DomainClinical trial registration no. NCT01091545© RSNA, 2021.

Project description:ObjectiveTo test the performance of an artificial intelligence-based computer-aided diagnosis (AI-CAD) designed for full-field digital mammography (FFDM) when applied to synthetic mammography (SM).Materials and methodsWe analyzed 501 women (mean age, 57 ± 11 years) who underwent preoperative mammography and breast cancer surgery. This cohort consisted of 1002 breasts, comprising 517 with cancer and 485 without. All patients underwent digital breast tomosynthesis (DBT) and FFDM during the preoperative workup. The SM is routinely reconstructed using DBT. Commercial AI-CAD (Lunit Insight MMG, version 1.1.7.2) was retrospectively applied to SM and FFDM to calculate the abnormality scores for each breast. The median abnormality scores were compared for the 517 breasts with cancer using the Wilcoxon signed-rank test. Calibration curves of abnormality scores were evaluated. The discrimination performance was analyzed using the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity using a 10% preset threshold. Sensitivity and specificity were further analyzed according to the mammographic and pathological characteristics. The results of SM and FFDM were compared.ResultsAI-CAD demonstrated a significantly lower median abnormality score (71% vs. 96%, P < 0.001) and poorer calibration performance for SM than for FFDM. SM exhibited lower sensitivity (76.2% vs. 82.8%, P < 0.001), higher specificity (95.5% vs. 91.8%, P < 0.001), and comparable AUC (0.86 vs. 0.87, P = 0.127) than FFDM. SM showed lower sensitivity than FFDM in asymptomatic breasts, dense breasts, ductal carcinoma in situ, T1, N0, and hormone receptor-positive/human epidermal growth factor receptor 2-negative cancers but showed higher specificity in non-cancerous dense breasts.ConclusionAI-CAD showed lower abnormality scores and reduced calibration performance for SM than for FFDM. Furthermore, the 10% preset threshold resulted in different discrimination performances for the SM. Given these limitations, off-label application of the current AI-CAD to SM should be avoided.

Project description:To compare the diagnostic performance of breast tomosynthesis versus supplemental mammography views in classification of masses, distortions, and asymmetries.Eight radiologists who specialized in breast imaging retrospectively reviewed 217 consecutively accrued lesions by using protocols that were HIPAA compliant and institutional review board approved in 182 patients aged 31-60 years (mean, 50 years) who underwent diagnostic mammography and tomosynthesis. The lesions in the cohort included 33% (72 of 217) cancers and 67% (145 of 217) benign lesions. Eighty-four percent (182 of 217) of the lesions were masses, 11% (25 of 217) were asymmetries, and 5% (10 of 217) were distortions that were initially detected at clinical examination in 8% (17 of 217), at mammography in 80% (173 of 217), at ultrasonography (US) in 11% (25 of 217), or at magnetic resonance imaging in 1% (2 of 217). Histopathologic examination established truth in 191 lesions, US revealed a cyst in 12 lesions, and 14 lesions had a normal follow-up. Each lesion was interpreted once with tomosynthesis and once with supplemental mammographic views; both modes included the mediolateral oblique and craniocaudal views in a fully crossed and balanced design by using a five-category Breast Imaging Reporting and Data System (BI-RADS) assessment and a probability-of-malignancy score. Differences between modes were analyzed with a generalized linear mixed model for BI-RADS-based sensitivity and specificity and with modified Obuchowski-Rockette approach for probability-of-malignancy-based area under the receiver operating characteristic (ROC) curve.Average probability-of-malignancy-based area under the ROC curve was 0.87 for tomosynthesis versus 0.83 for supplemental views (P < .001). With tomosynthesis, the false-positive rate decreased from 85% (989 of 1160) to 74% (864 of 1160) (P < .01) for cases that were rated BI-RADS category 3 or higher and from 57% (663 of 1160) to 48% (559 of 1160) for cases rated BI-RADS category 4 or 5 (P < .01), without a meaningful change in sensitivity. With tomosynthesis, more cancers were classified as BI-RADS category 5 (39% [226 of 576] vs 33% [188 of 576]; P = .017) without a decrease in specificity.Tomosynthesis significantly improved diagnostic accuracy for noncalcified lesions compared with supplemental mammographic views.

Project description:ImportanceA high proportion of suspicious pigmented skin lesions referred for investigation are benign. Techniques to improve the accuracy of melanoma diagnoses throughout the patient pathway are needed to reduce the pressure on secondary care and pathology services.ObjectiveTo determine the accuracy of an artificial intelligence algorithm in identifying melanoma in dermoscopic images of lesions taken with smartphone and digital single-lens reflex (DSLR) cameras.Design, setting, and participantsThis prospective, multicenter, single-arm, masked diagnostic trial took place in dermatology and plastic surgery clinics in 7 UK hospitals. Dermoscopic images of suspicious and control skin lesions from 514 patients with at least 1 suspicious pigmented skin lesion scheduled for biopsy were captured on 3 different cameras. Data were collected from January 2017 to July 2018. Clinicians and the Deep Ensemble for Recognition of Malignancy, a deterministic artificial intelligence algorithm trained to identify melanoma in dermoscopic images of pigmented skin lesions using deep learning techniques, assessed the likelihood of melanoma. Initial data analysis was conducted in September 2018; further analysis was conducted from February 2019 to August 2019.InterventionsClinician and algorithmic assessment of melanoma.Main outcomes and measuresArea under the receiver operating characteristic curve (AUROC), sensitivity, and specificity of the algorithmic and specialist assessment, determined using histopathology diagnosis as the criterion standard.ResultsThe study population of 514 patients included 279 women (55.7%) and 484 white patients (96.8%), with a mean (SD) age of 52.1 (18.6) years. A total of 1550 images of skin lesions were included in the analysis (551 [35.6%] biopsied lesions; 999 [64.4%] control lesions); 286 images (18.6%) were used to train the algorithm, and a further 849 (54.8%) images were missing or unsuitable for analysis. Of the biopsied lesions that were assessed by the algorithm and specialists, 125 (22.7%) were diagnosed as melanoma. Of these, 77 (16.7%) were used for the primary analysis. The algorithm achieved an AUROC of 90.1% (95% CI, 86.3%-94.0%) for biopsied lesions and 95.8% (95% CI, 94.1%-97.6%) for all lesions using iPhone 6s images; an AUROC of 85.8% (95% CI, 81.0%-90.7%) for biopsied lesions and 93.8% (95% CI, 91.4%-96.2%) for all lesions using Galaxy S6 images; and an AUROC of 86.9% (95% CI, 80.8%-93.0%) for biopsied lesions and 91.8% (95% CI, 87.5%-96.1%) for all lesions using DSLR camera images. At 100% sensitivity, the algorithm achieved a specificity of 64.8% with iPhone 6s images. Specialists achieved an AUROC of 77.8% (95% CI, 72.5%-81.9%) and a specificity of 69.9%.Conclusions and relevanceIn this study, the algorithm demonstrated an ability to identify melanoma from dermoscopic images of selected lesions with an accuracy similar to that of specialists.

Project description:Background While digital breast tomosynthesis (DBT) is rapidly replacing digital mammography (DM) in breast cancer screening, the potential of DBT density measures for breast cancer risk assessment remains largely unexplored. Purpose To compare associations of breast density estimates from DBT and DM with breast cancer. Materials and Methods This retrospective case-control study used contralateral DM/DBT studies from women with unilateral breast cancer and age- and ethnicity-matched controls (September 19, 2011-January 6, 2015). Volumetric percent density (VPD%) was estimated from DBT using previously validated software. For comparison, the publicly available Laboratory for Individualized Breast Radiodensity Assessment software package, or LIBRA, was used to estimate area-based percent density (APD%) from raw and processed DM images. The commercial Quantra and Volpara software packages were applied to raw DM images to estimate VPD% with use of physics-based models. Density measures were compared by using Spearman correlation coefficients (r), and conditional logistic regression was performed to examine density associations (odds ratios [OR]) with breast cancer, adjusting for age and body mass index. Results A total of 132 women diagnosed with breast cancer (mean age ± standard deviation [SD], 60 years ± 11) and 528 controls (mean age, 60 years ± 11) were included. Moderate correlations between DBT and DM density measures (r = 0.32-0.75; all P < .001) were observed. Volumetric density estimates calculated from DBT (OR, 2.3 [95% CI: 1.6, 3.4] per SD for VPD%DBT) were more strongly associated with breast cancer than DM-derived density for both APD% (OR, 1.3 [95% CI: 0.9, 1.9] [P < .001] and 1.7 [95% CI: 1.2, 2.3] [P = .004] per SD for LIBRA raw and processed data, respectively) and VPD% (OR, 1.6 [95% CI: 1.1, 2.4] [P = .01] and 1.7 [95% CI: 1.2, 2.6] [P = .04] per SD for Volpara and Quantra, respectively). Conclusion The associations between quantitative breast density estimates and breast cancer risk are stronger for digital breast tomosynthesis compared with digital mammography. © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Yaffe in this issue.

Dataset Information

A Competition, Benchmark, Code, and Data for Using Artificial Intelligence to Detect Lesions in Digital Breast Tomosynthesis.

Importance

Objectives

Design, setting, and participants

Main outcomes and measures

Results

Conclusions and relevance

Publications

A Competition, Benchmark, Code, and Data for Using Artificial Intelligence to Detect Lesions in Digital Breast Tomosynthesis.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets