Dataset Information

Algorithmic transparency and interpretability measures improve radiologists' performance in BI-RADS 4 classification.

ABSTRACT:

Objective

To evaluate the perception of different types of AI-based assistance and the interaction of radiologists with the algorithm's predictions and certainty measures.

Methods

In this retrospective observer study, four radiologists were asked to classify Breast Imaging-Reporting and Data System 4 (BI-RADS4) lesions (n = 101 benign, n = 99 malignant). The effect of different types of AI-based assistance (occlusion-based interpretability map, classification, and certainty) on the radiologists' performance (sensitivity, specificity, questionnaire) were measured. The influence of the Big Five personality traits was analyzed using the Pearson correlation.

Results

Diagnostic accuracy was significantly improved by AI-based assistance (an increase of 2.8% ± 2.3%, 95 %-CI 1.5 to 4.0 %, p = 0.045) and trust in the algorithm was generated primarily by the certainty of the prediction (100% of participants). Different human-AI interactions were observed ranging from nearly no interaction to humanization of the algorithm. High scores in neuroticism were correlated with higher persuasibility (Pearson's r = 0.98, p = 0.02), while higher consciousness and change of accuracy showed an inverse correlation (Pearson's r = -0.96, p = 0.04).

Conclusion

Trust in the algorithm's performance was mostly dependent on the certainty of the predictions in combination with a plausible heatmap. Human-AI interaction varied widely and was influenced by personality traits.

Key points

• AI-based assistance significantly improved the diagnostic accuracy of radiologists in classifying BI-RADS 4 mammography lesions. • Trust in the algorithm's performance was mostly dependent on the certainty of the prediction in combination with a reasonable heatmap. • Personality traits seem to influence human-AI collaboration. Radiologists with specific personality traits were more likely to change their classification according to the algorithm's prediction than others.

SUBMITTER: Jungmann F

PROVIDER: S-EPMC9935738 | biostudies-literature | 2023 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Algorithmic transparency and interpretability measures improve radiologists' performance in BI-RADS 4 classification.

Jungmann Friederike F Ziegelmayer Sebastian S Lohoefer Fabian K FK Metz Stephan S Müller-Leisse Christina C Englmaier Maximilian M Makowski Marcus R MR Kaissis Georgios A GA Braren Rickmer F RF

European radiology 20221025 3

<h4>Objective</h4>To evaluate the perception of different types of AI-based assistance and the interaction of radiologists with the algorithm's predictions and certainty measures.<h4>Methods</h4>In this retrospective observer study, four radiologists were asked to classify Breast Imaging-Reporting and Data System 4 (BI-RADS4) lesions (n = 101 benign, n = 99 malignant). The effect of different types of AI-based assistance (occlusion-based interpretability map, classification, and certainty) on th ...[more]

PMID: 36282311

Similar Datasets

Project description:Background Significant differences exist in the classification outcomes for radiologists using ultrasonography-based Breast Imaging Reporting and Data Systems for diagnosing category 3–5 (BI-RADS 3–5) breast nodules, due to a lack of clear and distinguishing image features. Consequently, this retrospective study investigated the improvement of BI-RADS 3–5 classification consistency using a transformer-based computer-aided diagnosis (CAD) model. Methods Independently, 5 radiologists performed BI-RADS annotations on 21,332 breast ultrasonographic images collected from 3,978 female patients from 20 clinical centers in China. All images were divided into training, validation, testing, and sampling sets. The trained transformer-based CAD model was then used to classify test images, for which sensitivity (SEN), specificity (SPE), accuracy (ACC), area under the curve (AUC), and calibration curve were evaluated. Variations in these metrics among the 5 radiologists were analyzed by referencing BI-RADS classification results for the sampling test set provided by CAD to determine whether classification consistency (the k value), SEN, SPE, and ACC could be improved. Results After the training set (11,238 images) and validation set (2,996 images) were learned by the CAD model, the classification ACC of the CAD model applied to the test set (7,098 images) was 94.89% in category 3, 96.90% in category 4A, 95.49% in category 4B, 92.28% in category 4C, and 95.45% in category 5 nodules. Based on pathological results, the AUC of the CAD model was 0.924 and the predicted probability of CAD was a little higher than the actual probability in the calibration curve. After referencing BI-RADS classification results, the adjustments were made to 1,583 nodules, of which 905 were classified to a lower category and 678 to a higher category in the sampling test set. As a result, the ACC (72.41–82.65%), SEN (32.73–56.98%), and SPE (82.46–89.26%) of the classification by each radiologist were significantly improved on average, with the consistency (k values) in almost all of them increasing to >0.6. Conclusions The radiologist’s classification consistency was markedly improved with almost all the k values increasing by a value greater than 0.6, and the diagnostic efficiency was also improved by approximately 24% (32.73% to 56.98%) and 7% (82.46% to 89.26%) for SEN and SPE, respectively, of the total classification on average. The transformer-based CAD model can help to improve the radiologist’s diagnostic efficacy and consistency with others in the classification of BI-RADS 3–5 nodules.

Project description:To evaluate the positive predictive values (PPVs) of Breast Imaging and Reporting Data Systems (BI-RADS) assessment categories for breast magnetic resonance (MR) imaging and to identify the BI-RADS MR imaging lesion features most predictive of cancer.This institutional review board-approved HIPAA-compliant prospective multicenter study was performed with written informed consent. Breast MR imaging studies of the contralateral breast in women with a recent diagnosis of breast cancer were prospectively evaluated. Contralateral breast MR imaging BI-RADS assessment categories, morphologic descriptors for foci, masses, non-masslike enhancement (NMLE), and kinetic features were assessed for predictive values for malignancy. PPV of each imaging characteristic of interest was estimated, and logistic regression analysis was used to examine the predictive ability of combinations of characteristics.Of 969 participants, 71.3% had a BI-RADS category 1 or 2 assessment; 10.9%, a BI-RADS category 3 assessment; 10.0%, a BI-RADS category 4 or 5 assessment; and 7.7%, a BI-RADS category 0 assessment on the basis of initial MR images. Thirty-one cancers were detected with MR imaging. Overall PPV for BI-RADS category 4 and 5 lesions was 0.278, with 17 cancers in patients with a BI-RADS category 4 lesion (PPV, 0.205) and 10 cancers in patients with a BI-RADS category 5 lesion (PPV, 0.714). Of the cancers, one was a focus, 17 were masses, and 13 were NMLEs. For masses, irregular shape, irregular margins, spiculated margins, and marked internal enhancement were most predictive of malignancy. For NMLEs, ductal, clumped, and reticular or dendritic enhancement were the features most frequently seen with malignancy. Kinetic enhancement features were less predictive of malignancy than were morphologic features.Standardized terminology of the BI-RADS lexicon enables quantification of the likelihood of malignancy for MR imaging-detected lesions through careful evaluation of lesion features. In particular, BI-RADS assessment categories and morphologic descriptors for masses and NMLE were useful in estimating the probability of cancer.

Project description:To evaluate the interobserver variability in descriptions of breast masses by dedicated breast imagers and radiology residents and determine how any differences in lesion description affect the performance of a computer-aided diagnosis (CAD) computer classification system.Institutional review board approval was obtained for this HIPAA-compliant study, and the requirement to obtain informed consent was waived. Images of 50 breast lesions were individually interpreted by seven dedicated breast imagers and 10 radiology residents, yielding 850 lesion interpretations. Lesions were described with use of 11 descriptors from the Breast Imaging Reporting and Data System, and interobserver variability was calculated with the Cohen κ statistic. Those 11 features were selected, along with patient age, and merged together by a linear discriminant analysis (LDA) classification model trained by using 1005 previously existing cases. Variability in the recommendations of the computer model for different observers was also calculated with the Cohen κ statistic.A significant difference was observed for six lesion features, and radiology residents had greater interobserver variability in their selection of five of the six features than did dedicated breast imagers. The LDA model accurately classified lesions for both sets of observers (area under the receiver operating characteristic curve = 0.94 for residents and 0.96 for dedicated imagers). Sensitivity was maintained at 100% for residents and improved from 98% to 100% for dedicated breast imagers. For residents, the computer model could potentially improve the specificity from 20% to 40% (P < .01) and the κ value from 0.09 to 0.53 (P < .001). For dedicated breast imagers, the computer model could increase the specificity from 34% to 43% (P = .16) and the κ value from 0.21 to 0.61 (P < .001).Among findings showing a significant difference, there was greater interobserver variability in lesion descriptions among residents; however, an LDA model using data from either dedicated breast imagers or residents yielded a consistently high performance in the differentiation of benign from malignant breast lesions, demonstrating potential for improving specificity and decreasing interobserver variability in biopsy recommendations.

Dataset Information

Algorithmic transparency and interpretability measures improve radiologists' performance in BI-RADS 4 classification.

Objective

Methods

Results

Conclusion

Key points

Publications

Algorithmic transparency and interpretability measures improve radiologists' performance in BI-RADS 4 classification.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets