Multi-modal transformer architecture for medical image analysis and automated report generation.

ABSTRACT: Medical practitioners examine medical images, such as X-rays, write reports based on the findings, and provide conclusive statements. Manual interpretation of the results and report generation by examiners are time-consuming processes that lead to potential delays in diagnosis. We propose an automated report generation model for medical images leveraging an encoder-decoder architecture. Our model utilizes transformer architectures, including Vision Transformer (ViT) and its variants like Data Efficient Image Transformer (DEiT) and BERT pre-training image transformer (BEiT), as an encoder. These transformers are adapted for processing to extract and gain visual information from medical images. Reports are transformed into text embeddings, and the Generative Pre-trained Transformer (GPT2) mo

SUBMITTER: Raminedi S 

PROVIDER: S-EPMC11336090 | biostudies-literature | 2024 Aug

REPOSITORIES: biostudies-literature