Dataset Information

Assessment of Expert-Level Automated Detection of Plasmodium falciparum in Digitized Thin Blood Smear Images.

ABSTRACT: Importance:Decades of effort have been devoted to establishing an automated microscopic diagnosis of malaria, but there are challenges in achieving expert-level performance in real-world clinical settings because publicly available annotated data for benchmark and validation are required. Objective:To assess an expert-level malaria detection algorithm using a publicly available benchmark image data set. Design, Setting, and Participants:In this diagnostic study, clinically validated malaria image data sets, the Taiwan Images for Malaria Eradication (TIME), were created by digitizing thin blood smears acquired from patients with malaria selected from the biobank of the Taiwan Centers for Disease Control from January 1, 2003, to December 31, 2018. These smear images were annotated by 4 clinical laboratory scientists who worked in medical centers in Taiwan and trained for malaria microscopic diagnosis at the national reference laboratory of the Taiwan Centers for Disease Control. With TIME, a convolutional neural network-based object detection algorithm was developed for identification of malaria-infected red blood cells. A diagnostic challenge using another independent data set within TIME was performed to compare the algorithm performance against that of human experts as clinical validation. Main Outcomes and Measures:Performance on detecting Plasmodium falciparum-infected blood cells was measured by average precision, and performance on detecting P falciparum infection at the image level was measured using sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). Results:The TIME data sets contained 8145 images of 36 blood smears from patients with suspected malaria (30 P falciparum-positive and 6 P falciparum-negative smears) that had reliable annotations. For clinical validation, the average precision was 0.885 for detecting P falciparum-infected blood cells and 0.838 for ring form. For detecting P falciparum infection on blood smear images, the algorithm had expert-level performance (sensitivity, 0.995; specificity, 0.900; AUC, 0.997 [95% CI, 0.993-0.999]), especially in detecting ring form (sensitivity, 0.968; specificity, 0.960; AUC, 0.995 [95% CI, 0.990-0.998]) compared with experienced microscopists (mean sensitivity, 0.995 [95% CI, 0.993-0.998]; mean specificity, 0.955 [95% CI, 0.885-1.000]). Conclusions and Relevance:The findings suggest that a clinically validated expert-level malaria detection algorithm can be developed by using reliable data sets.

SUBMITTER: Kuo PC

PROVIDER: S-EPMC7049085 | biostudies-literature | 2020 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessment of Expert-Level Automated Detection of Plasmodium falciparum in Digitized Thin Blood Smear Images.

Kuo Po-Chen PC Cheng Hao-Yuan HY Chen Pi-Fang PF Liu Yu-Lun YL Kang Martin M Kuo Min-Chu MC Hsu Shih-Fen SF Lu Hsin-Jung HJ Hong Stefan S Su Chan-Hung CH Liu Ding-Ping DP Tu Yi-Chin YC Chuang Jen-Hsiang JH

JAMA network open 20200205 2

<h4>Importance</h4>Decades of effort have been devoted to establishing an automated microscopic diagnosis of malaria, but there are challenges in achieving expert-level performance in real-world clinical settings because publicly available annotated data for benchmark and validation are required.<h4>Objective</h4>To assess an expert-level malaria detection algorithm using a publicly available benchmark image data set.<h4>Design, setting, and participants</h4>In this diagnostic study, clinically ...[more]

PMID: 32108895

Similar Datasets

Project description:BackgroundThe conventional method for the diagnosis of malaria parasites is the microscopic examination of stained blood films, which is time consuming and requires expertise. We introduce computer-based image segmentation and life stage classification with a random forest classifier. Segmentation and stage classification are performed on a large dataset of malaria parasites with ground truth labels provided by experts.MethodsWe made use of Giemsa stained images obtained from the blood of 16 patients infected with Plasmodium falciparum. Experts labeled the parasite types from each of the images. We applied a two-step approach: image segmentation followed by life stage classification. In segmentation, we classified each pixel as a parasite or non-parasite pixel using a random forest classifier. Performance was evaluated with classification accuracy, Dice coefficient and free-response receiver operating characteristic (FROC) analysis. In life stage classification, we classified each of the segmented objects into one of 8 classes: 6 parasite life stages, early ring, late ring or early trophozoite, mid trophozoite, early schizont, late schizont or segmented, and two other classes, white blood cell or debris.ResultsOur segmentation method gives an average cross-validated Dice coefficient of 0.82 which is a 13% improvement compared to the Otsu method. The Otsu method achieved a True Positive Fraction (TPF) of 0.925 at the expense of a False Positive Rate (FPR) of 2.45. At the same TPF of 0.925, our method achieved an FPR of 0.92, an improvement by more than a factor two. We find that inclusion of average intensity of the whole image as feature for the random forest considerably improves segmentation performance. We obtain an overall accuracy of 58.8% when classifying all life stages. Stages are mostly confused with their neighboring stages. When we reduce the life stages to ring, trophozoite and schizont only, we obtain an accuracy of 82.7%.ConclusionPixel classification gives better segmentation performance than the conventional Otsu method. Effects of staining and background variations can be reduced with the inclusion of average intensity features. The proposed method and data set can be used in the development of automatic tools for the detection and stage classification of malaria parasites. The data set is publicly available as a benchmark for future studies.

Project description:BackgroundThe ability of mature forms of Plasmodium falciparum infected erythrocytes to bind to a range of host receptors including those displayed on endothelial cells has been associated with the pathology of this infection. Investigations into this adhesive phenomenon have used protein and cell-based adhesion assays to quantify the ability of infected red blood cells to bind. These adhesion assays tend to have relatively high inherent variability and so require multiple experiments in order to provide good quantitation. This means that investigators doing these experiments must count many fields of adherent parasites, a task that is time-consuming and laborious. To address this issue and to facilitate cytoadherence research, developed automated protocols were developed for counting parasite adhesion.MethodsParasite adhesion assays were mainly carried out under static conditions using purified receptors, which is the simplest form of these assays and is translatable to the field. Two different software platforms were used, one commercial (Image Pro-Plus (Media Cybernetics)) and one available in the public domain (ImageSXM) based on the freely available NIH Image software. The adhesion assays were performed and parasite binding quantified using standard manual techniques. Images were also captured using video microscopy and analysed using the two automated systems. The results generated by each system were compared using the Bland and Altman method for assessing the agreement between two methods.ResultsBoth automated counting programs showed concordance compared to the 'gold standard' manual counting within the normal range of adhesion seen with these assays, although the ImageSXM technique had some systematic bias. There was some fall-off in accuracy at very high parasite densities, but this can be resolved through good design of the experiments. Cell based assays were also used as inputs to one of the automated systems (ImageSXM) and produced variable, but encouraging, results.ConclusionsThe automated counting programs are an accurate and practical way of quantifying static parasite binding assays to purified proteins. They are less accurate when applied to cell based systems, but can still provide a reasonable level of accuracy to give a semi-quantitative readout.

Project description:Malaria remains a global health problem, with 247 million cases and 619,000 deaths in 2021. Diagnosis of Plasmodium species is important for administering the appropriate treatment. The gold-standard diagnosis for accurate species identification remains the thin blood smear. Nevertheless, this method is time-consuming and requires highly skilled and trained microscopists. To overcome these issues, new diagnostic tools based on deep learning are emerging. This study aimed to evaluate the performances of a real-time detection transformer (RT-DETR) object detection algorithm to discriminate Plasmodium species on thin blood smear images. The algorithm was trained and validated on a data set consisting in 24,720 images from 475 thin blood smears corresponding to 2,002,597 labels. Performances were calculated with a test data set of 4,508 images from 170 smears corresponding to 358,825 labels coming from six French university hospitals. At the patient level, the RT-DETR algorithm exhibited an overall accuracy of 79.4% (135/170) with a recall of 74% (40/54) and 81.9% (95/116) for negative and positive smears, respectively. Among Plasmodium-positive smears, the global accuracy was 82.7% (91/110) with a recall of 90% (38/42), 81.8% (18/22), and 76.1% (35/46) for P. falciparum, P. malariae, and P. ovale/vivax, respectively. The RT-DETR model achieved a World Health Organization (WHO) competence level 2 for species identification. Besides, the RT-DETR algorithm may be run in real-time on low-cost devices such as a smartphone and could be suitable for deployment in low-resource setting areas lacking microscopy experts.IMPORTANCEMalaria remains a global health problem, with 247 million cases and 619,000 deaths in 2021. Diagnosis of Plasmodium species is important for administering the appropriate treatment. The gold-standard diagnosis for accurate species identification remains the thin blood smear. Nevertheless, this method is time-consuming and requires highly skilled and trained microscopists. To overcome these issues, new diagnostic tools based on deep learning are emerging. This study aimed to evaluate the performances of a real-time detection transformer (RT-DETR) object detection algorithm to discriminate Plasmodium species on thin blood smear images. Performances were calculated with a test data set of 4,508 images from 170 smears coming from six French university hospitals. The RT-DETR model achieved a World Health Organization (WHO) competence level 2 for species identification. Besides, the RT-DETR algorithm may be run in real-time on low-cost devices and could be suitable for deployment in low-resource setting areas.

Project description:BackgroundMalaria parasitemia is commonly used as a measurement of the amount of parasites in the patient's blood and a crucial indicator for the degree of infection. Manual evaluation of Giemsa-stained thin blood smears under the microscope is onerous, time consuming and subject to human error. Although automatic assessments can overcome some of these problems the available methods are currently limited by their inability to evaluate cases that deviate from a chosen "standard" model.ResultsIn this study reliable parasitemia counts were achieved even for sub-standard smear and image quality. The outcome was assessed through comparisons with manual evaluations of more than 200 sample smears and related to the complexity of cell overlaps. On average an estimation error of less than 1% with respect to the average of manually obtained parasitemia counts was achieved. In particular the results from the proposed approach are generally within one standard deviation of the counts provided by a comparison group of malariologists yielding a correlation of 0.97. Variations occur mainly for blurred out-of-focus imagery exhibiting larger degrees of cell overlaps in clusters of erythrocytes. The assessment was also carried out in terms of precision and recall and combined in the F-measure providing results generally in the range of 92% to 97% for a variety of smears. In this context the observed trade-off relation between precision and recall guaranteed stable results. Finally, relating the F-measure with the degree of cell overlaps, showed that up to 50% total cell overlap can be tolerated if the smear image is well-focused and the smear itself adequately stained.ConclusionThe automatic analysis has proven to be comparable with manual evaluations in terms of accuracy. Moreover, the test results have shown that the proposed comparison-based approach, by exploiting the interrelation between different images and color channels, has successfully overcome most of the inherent limitations possibly occurring during the sample preparation and image acquisition phase. Eventually, this can be seen as an opportunity for developing low-cost solutions for mass screening.

Project description:The letter-guided naming fluency task is a measure of an individual's executive function and working memory. This study employed a novel, automated, quantifiable, and reproducible method to investigate how language characteristics of words produced during a fluency task are related to fluency performance, inter-word response time (RT), and over task duration using digitized F-letter-guided fluency recordings produced by 76 young healthy participants. Our automated algorithm counted the number of correct responses from the transcripts of the F-letter fluency data, and individual words were rated for concreteness, ambiguity, frequency, familiarity, and age of acquisition (AoA). Using a forced aligner, the transcripts were automatically aligned with the corresponding audio recordings. We measured inter-word RT, word duration, and word start time from the forced alignments. Articulation rate was also computed. Phonetic and semantic distances between two consecutive F-letter words were measured. We found that total F-letter score was significantly correlated with the mean values of word frequency, familiarity, AoA, word duration, phonetic similarity, and articulation rate; total score was also correlated with an individual's standard deviation of AoA, familiarity, and phonetic similarity. RT was negatively correlated with frequency and ambiguity of F-letter words and was positively correlated with AoA, number of phonemes, and phonetic and semantic distances. Lastly, the frequency, ambiguity, AoA, number of phonemes, and semantic distance of words produced significantly changed over time during the task. The method employed in this paper demonstrates the successful implementation of our automated language processing pipelines in a standardized neuropsychological task. This novel approach captures subtle and rich language characteristics during test performance that enhance informativeness and cannot be extracted manually without massive effort. This work will serve as the reference for letter-guided category fluency production similarly acquired in neurodegenerative patients.

Dataset Information

Assessment of Expert-Level Automated Detection of Plasmodium falciparum in Digitized Thin Blood Smear Images.

Publications

Assessment of Expert-Level Automated Detection of Plasmodium falciparum in Digitized Thin Blood Smear Images.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets