Dataset Information

Validation of an established TW3 artificial intelligence bone age assessment system: a prospective, multicenter, confirmatory study.

ABSTRACT:

Background

In 2020, our center established a Tanner-Whitehouse 3 (TW3) artificial intelligence (AI) system using a convolutional neural network (CNN), which was built upon 9059 radiographs. However, the system, upon which our study is based, lacked a gold standard for comparison and had not undergone thorough evaluation in different working environments.

Methods

To further verify the applicability of the AI system in clinical bone age assessment (BAA) and to enhance the accuracy and homogeneity of BAA, a prospective multi-center validation was conducted. This study utilized 744 left-hand radiographs of patients, ranging from 1 to 20 years of age, with 378 boys and 366 girls. These radiographs were obtained from nine different children's hospitals between August and December 2020. The BAAs were performed using the TW3 AI system and were also reviewed by experienced reviewers. Bone age accuracy within 1 year, root mean square error (RMSE), and mean absolute error (MAE) were statistically calculated to evaluate the accuracy. Kappa test and Bland-Altman (B-A) plot were conducted to measure the diagnostic consistency.

Results

The system exhibited a high level of performance, producing results that closely aligned with those of the reviewers. It achieved a RMSE of 0.52 years and an accuracy of 94.55% for the radius, ulna, and short bones series. When assessing the carpal series of bones, the system achieved a RMSE of 0.85 years and an accuracy of 80.38%. Overall, the system displayed satisfactory accuracy and RMSE, particularly in patients over 7 years old. The system excelled in evaluating the carpal bone age of patients aged 1-6. Both the Kappa test and B-A plot demonstrated substantial consistency between the system and the reviewers, although the model encountered challenges in consistently distinguishing specific bones, such as the capitate. Furthermore, the system's performance proved acceptable across different genders and age groups, as well as radiography instruments.

Conclusions

In this multi-center validation, the system showcased its potential to enhance the efficiency and consistency of healthy delivery, ultimately resulting in improved patient outcomes and reduced healthcare costs.

SUBMITTER: Liu Y

PROVIDER: S-EPMC10784042 | biostudies-literature | 2024 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Validation of an established TW3 artificial intelligence bone age assessment system: a prospective, multicenter, confirmatory study.

Liu Yanqi Y Ouyang Liujian L Wu Wei W Zhou Xuelian X Huang Ke K Wang Zhihua Z Song Cui C Chen Qiuli Q Su Zhe Z Zheng Rongxiu R Wei Ying Y Lu Wei W Wu Wei W Liu Yang Y Yan Ziye Z Wu Zhaoyuan Z Fan Jitao J Zhou Mingzhi M Fu Junfen J

Quantitative imaging in medicine and surgery 20231028 1

<h4>Background</h4>In 2020, our center established a Tanner-Whitehouse 3 (TW3) artificial intelligence (AI) system using a convolutional neural network (CNN), which was built upon 9059 radiographs. However, the system, upon which our study is based, lacked a gold standard for comparison and had not undergone thorough evaluation in different working environments.<h4>Methods</h4>To further verify the applicability of the AI system in clinical bone age assessment (BAA) and to enhance the accuracy a ...[more]

PMID: 38223047

Similar Datasets

Project description:Background: Manual bone age assessment (BAA) is associated with longer interpretation time and higher cost and variability, thus posing challenges in areas with restricted medical facilities, such as the high-altitude Tibetan Plateau. The application of artificial intelligence (AI) for automating BAA could facilitate resolving this issue. This study aimed to develop an AI-based BAA model for Han and Tibetan children. Methods: A model named "EVG-BANet" was trained using three datasets, including the Radiology Society of North America (RSNA) dataset (training set n = 12611, validation set n = 1425, and test set n = 200), the Radiological Hand Pose Estimation (RHPE) dataset (training set n = 5491, validation set n = 713, and test set n = 79), and a self-established local dataset [training set n = 825 and test set n = 351 (Han n = 216 and Tibetan n = 135)]. An open-access state-of-the-art model BoNet was used for comparison. The accuracy and generalizability of the two models were evaluated using the abovementioned three test sets and an external test set (n = 256, all were Tibetan). Mean absolute difference (MAD) and accuracy within 1 year were used as indicators. Bias was evaluated by comparing the MAD between the demographic groups. Results: EVG-BANet outperformed BoNet in the MAD on the RHPE test set (0.52 vs. 0.63 years, p < 0.001), the local test set (0.47 vs. 0.62 years, p < 0.001), and the external test set (0.53 vs. 0.66 years, p < 0.001) and exhibited a comparable MAD on the RSNA test set (0.34 vs. 0.35 years, p = 0.934). EVG-BANet achieved accuracy within 1 year of 97.7% on the local test set (BoNet 90%, p < 0.001) and 89.5% on the external test set (BoNet 85.5%, p = 0.066). EVG-BANet showed no bias in the local test set but exhibited a bias related to chronological age in the external test set. Conclusion: EVG-BANet can accurately predict the bone age (BA) for both Han children and Tibetan children living in the Tibetan Plateau with limited healthcare facilities.

Project description:BackgroundMobile health (mHealth) consumer applications (apps) have been integrated with deep learning for skin cancer risk assessments. However, prospective validation of these apps is lacking.ObjectivesTo identify the diagnostic accuracy of an app integrated with a convolutional neural network for the detection of premalignant and malignant skin lesions.MethodsWe performed a prospective multicenter diagnostic accuracy study of a CE-marked mHealth app from January 1 until August 31, 2020, among adult patients with at least one suspicious skin lesion. Skin lesions were assessed by the app on an iOS or Android device after clinical diagnosis and before obtaining histopathology. The app outcome was compared to the histopathological diagnosis, or if not available, the clinical diagnosis by a dermatologist. The primary outcome was the sensitivity and specificity of the app to detect premalignant and malignant skin lesions. Subgroup analyses were conducted for different smartphone types, the lesion's origin, indication for dermatological consultation, and lesion location.ResultsIn total, 785 lesions, including 418 suspicious and 367 benign control lesions, among 372 patients (50.8% women) with a median age of 71 years were included. The app performed at an overall 86.9% (95% CI 82.3-90.7) sensitivity and 70.4% (95% CI 66.2-74.3) specificity. The sensitivity was significantly higher on the iOS device compared to the Android device (91.0 vs. 83.0%; p = 0.02). Specificity calculated on benign control lesions was significantly higher than suspicious skin lesions (80.1 vs. 45.5%; p < 0.001). Sensitivity was higher in skin fold areas compared to smooth skin areas (92.9 vs. 84.2%; p = 0.01), while the specificity was higher for lesions in smooth skin areas (72.0 vs. 56.6%; p = 0.02).ConclusionThe diagnostic accuracy of the mHealth app is far from perfect, but is potentially promising to empower patients to self-assess skin lesions before consulting a health care professional. An additional prospective validation study, particularly for suspicious pigmented skin lesions, is warranted. Furthermore, studies investigating mHealth implementation in the lay population are needed to demonstrate the impact on health care systems.

Dataset Information

Validation of an established TW3 artificial intelligence bone age assessment system: a prospective, multicenter, confirmatory study.

Background

Methods

Results

Conclusions

Publications

Validation of an established TW3 artificial intelligence bone age assessment system: a prospective, multicenter, confirmatory study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets