Dataset Information

Comparative analysis of machine learning algorithms for computer-assisted reporting based on fully automated cross-lingual RadLex mappings.

ABSTRACT: Computer-assisted reporting (CAR) tools were suggested to improve radiology report quality by context-sensitively recommending key imaging biomarkers. However, studies evaluating machine learning (ML) algorithms on cross-lingual ontological (RadLex) mappings for developing embedded CAR algorithms are lacking. Therefore, we compared ML algorithms developed on human expert-annotated features against those developed on fully automated cross-lingual (German to English) RadLex mappings using 206 CT reports of suspected stroke. Target label was whether the Alberta Stroke Programme Early CT Score (ASPECTS) should have been provided (yes/no:154/52). We focused on probabilistic outputs of ML-algorithms including tree-based methods, elastic net, support vector machines (SVMs) and fastText (linear classifier), which were evaluated in the same 5 × fivefold nested cross-validation framework. This allowed for model stacking and classifier rankings. Performance was evaluated using calibration metrics (AUC, brier score, log loss) and -plots. Contextual ML-based assistance recommending ASPECTS was feasible. SVMs showed the highest accuracies both on human-extracted- (87%) and RadLex features (findings:82.5%; impressions:85.4%). FastText achieved the highest accuracy (89.3%) and AUC (92%) on impressions. Boosted trees fitted on findings had the best calibration profile. Our approach provides guidance for choosing ML classifiers for CAR tools in fully automated and language-agnostic fashion using bag-of-RadLex terms on limited expert-labelled training data.

SUBMITTER: Maros ME

PROVIDER: S-EPMC7970897 | biostudies-literature | 2021 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Comparative analysis of machine learning algorithms for computer-assisted reporting based on fully automated cross-lingual RadLex mappings.

Maros Máté E ME Cho Chang Gyu CG Junge Andreas G AG Kämpgen Benedikt B Saase Victor V Siegel Fabian F Trinkmann Frederik F Ganslandt Thomas T Groden Christoph C Wenz Holger H

Scientific reports 20210309 1

Computer-assisted reporting (CAR) tools were suggested to improve radiology report quality by context-sensitively recommending key imaging biomarkers. However, studies evaluating machine learning (ML) algorithms on cross-lingual ontological (RadLex) mappings for developing embedded CAR algorithms are lacking. Therefore, we compared ML algorithms developed on human expert-annotated features against those developed on fully automated cross-lingual (German to English) RadLex mappings using 206 CT r ...[more]

PMID: 33750857

Similar Datasets

Project description:Computer-Assisted Semen Analysis (CASA) enables reliable analysis of semen images, and is designed to process large number of images with high consistency, accuracy, and repeatability. Design and testing of CASA algorithms can be accelerated greatly if reliable simulations of semen images under a variety of conditions and sample quality modes are available. Using life-like simulation of semen images can quantify the performance of existing and proposed CASA algorithms, since the parameters of the simulated image are known and controllable. We present simulation models for sperm cell image and swimming modes observed in real 2D (top-down) images of sperm cells in laboratory specimen. The models simulate human sperm using four (4) types of swimming, namely linear mean, circular, hyperactive, and immotile (or dead). The simulation models are used in studying algorithms for segmentation, localization, and tracking of sperm cells. Several segmentation and localization algorithms were tested under varying levels of noise, and then compared using precision, recall, and the optimal subpattern assignment (OSPA) metric. Images of real human semen sample were used to validate the segmentation and localization observations obtained from simulations. An example is given of sperm cell tracking on simulated semen images of cells using the different tracking algorithms (nearest neighbor (NN), global nearest neighbor (GNN), probabilistic data association filter (PDAF), and joint probabilistic data association filter (JPDAF)). Tracking performance was evaluated through multi-object tracking precision (MOTP) and multi-object tracking accuracy (MOTA). Simulation models enable objective assessments of semen image processing algorithms. We demonstrate the use of a new simulation tool to assess and compare segmentation, localization, and tracking methods. The simulation software allows testing along a large spectrum of parameter values that control the appearance and behavior of simulated semen images. Users can generate scenarios of different characteristics and assess the effectiveness of different CASA algorithms in these environments. The simulation was used to assess and compare algorithms for segmentation and tracking of sperm cells in semen images.

Project description:ObjectivesSuccessful lung cancer screening delivery requires sensitive, timely reporting of low-dose computed tomography (LDCT) scans, placing a demand on radiology resources. Trained non-radiologist readers and computer-assisted detection (CADe) software may offer strategies to optimise the use of radiology resources without loss of sensitivity. This report examines the accuracy of trained reporting radiographers using CADe support to report LDCT scans performed as part of the Lung Screen Uptake Trial (LSUT).MethodsIn this observational cohort study, two radiographers independently read all LDCT performed within LSUT and reported on the presence of clinically significant nodules and common incidental findings (IFs), including recommendations for management. Reports were compared against a 'reference standard' (RS) derived from nodules identified by study radiologists without CADe, plus consensus radiologist review of any additional nodules identified by the radiographers.ResultsA total of 716 scans were included, 158 of which had one or more clinically significant pulmonary nodules as per our RS. Radiographer sensitivity against the RS was 68-73.7%, with specificity of 92.1-92.7%. Sensitivity for detection of proven cancers diagnosed from the baseline scan was 83.3-100%. The spectrum of IFs exceeded what could reasonably be covered in radiographer training.ConclusionOur findings highlight the complexity of LDCT reporting requirements, including the limitations of CADe and the breadth of IFs. We are unable to recommend CADe-supported radiographers as a sole reader of LDCT scans, but propose potential avenues for further research including initial triage of abnormal LDCT or reporting of follow-up surveillance scans.Key points• Successful roll-out of mass screening programmes for lung cancer depends on timely, accurate CT scan reporting, placing a demand on existing radiology resources. • This observational cohort study examines the accuracy of trained radiographers using computer-assisted detection (CADe) software to report lung cancer screening CT scans, as a potential means of supporting reporting workflows in LCS programmes. • CADe-supported radiographers were less sensitive than radiologists at identifying clinically significant pulmonary nodules, but had a low false-positive rate and good sensitivity for detection of confirmed cancers.

Project description:ObjectiveTo develop two automated computer algorithms to extract information from clinical notes, and to generate three cohorts of patients (disease improvement, disease progression, and no disease change) to track periodontal disease (PD) change over time using longitudinal electronic dental records (EDR).MethodsWe conducted a retrospective study of 28,908 patients who received a comprehensive oral evaluation between 1 January 2009, and 31 December 2014, at Indiana University School of Dentistry (IUSD) clinics. We utilized various Python libraries, such as Pandas, TensorFlow, and PyTorch, and a natural language tool kit to develop and test computer algorithms. We tested the performance through a manual review process by generating a confusion matrix. We calculated precision, recall, sensitivity, specificity, and accuracy to evaluate the performances of the algorithms. Finally, we evaluated the density of longitudinal EDR data for the following follow-up times: (1) None; (2) Up to 5 years; (3) > 5 and ≤ 10 years; and (4) >10 and ≤ 15 years.ResultsThirty-four percent (n = 9954) of the study cohort had up to five years of follow-up visits, with an average of 2.78 visits with periodontal charting information. For clinician-documented diagnoses from clinical notes, 42% of patients (n = 5562) had at least two PD diagnoses to determine their disease change. In this cohort, with clinician-documented diagnoses, 72% percent of patients (n = 3919) did not have a disease status change between their first and last visits, 669 (13%) patients' disease status progressed, and 589 (11%) patients' disease improved.ConclusionsThis study demonstrated the feasibility of utilizing longitudinal EDR data to track disease changes over 15 years during the observation study period. We provided detailed steps and computer algorithms to clean and preprocess the EDR data and generated three cohorts of patients. This information can now be utilized for studying clinical courses using artificial intelligence and machine learning methods.

Dataset Information

Comparative analysis of machine learning algorithms for computer-assisted reporting based on fully automated cross-lingual RadLex mappings.

Publications

Comparative analysis of machine learning algorithms for computer-assisted reporting based on fully automated cross-lingual RadLex mappings.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets