Dataset Information

Multimodal-GuideNet: Gaze-Probe Bidirectional Guidance in Obstetric Ultrasound Scanning.

ABSTRACT: Eye trackers can provide visual guidance to sonographers during ultrasound (US) scanning. Such guidance is potentially valuable for less experienced operators to improve their scanning skills on how to manipulate the probe to achieve the desired plane. In this paper, a multimodal guidance approach (Multimodal-GuideNet) is proposed to capture the stepwise dependency between a real-world US video signal, synchronized gaze, and probe motion within a unified framework. To understand the causal relationship between gaze movement and probe motion, our model exploits multitask learning to jointly learn two related tasks: predicting gaze movements and probe signals that an experienced sonographer would perform in routine obstetric scanning. The two tasks are associated by a modality-aware spatial graph to detect the co-occurrence among the multi-modality inputs and share useful cross-modal information. Instead of a deterministic scanning path, Multimodal-GuideNet allows for scanning diversity by estimating the probability distribution of real scans. Experiments performed with three typical obstetric scanning examinations show that the new approach outperforms single-task learning for both probe motion guidance and gaze movement prediction. Multimodal-GuideNet also provides a visual guidance signal with an error rate of less than 10 pixels for a 224 × 288 US image.

SUBMITTER: Men Q

PROVIDER: S-EPMC7614062 | biostudies-literature | 2022 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Multimodal-GuideNet: Gaze-Probe Bidirectional Guidance in Obstetric Ultrasound Scanning.

Men Qianhui Q Teng Clare C Drukker Lior L Papageorghiou Aris T AT Noble J Alison JA

Medical image computing and computer-assisted intervention : MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention 20220901

Eye trackers can provide visual guidance to sonographers during ultrasound (US) scanning. Such guidance is potentially valuable for less experienced operators to improve their scanning skills on how to manipulate the probe to achieve the desired plane. In this paper, a multimodal guidance approach (Multimodal-GuideNet) is proposed to capture the stepwise dependency between a real-world US video signal, synchronized gaze, and probe motion within a unified framework. To understand the causal relat ...[more]

PMID: 36649382

Similar Datasets

Project description:BackgroundConvex probe endobronchial ultrasound images can reflect the morphology, blood flow status and stiffness of the lesions. Endobronchial ultrasound multimodal imaging has great value for the diagnosis of intrathoracic lymph nodes. This study aimed to analyze the application of endobronchial ultrasound multimodal imaging on lung lesions.MethodsPatients undergoing endobronchial ultrasound-guided transbronchial needle aspiration in Shanghai Chest Hospital from July 2018 to December 2019 were retrospectively enrolled. Nine grayscale features (long and short axes, margin, shape, lobulation sign, echogenicity, necrosis, liquefaction, calcification, and air-bronchogram), blood flow volume and elastography five-score method were analyzed to explore the best diagnostic method. The gold standard for diagnosing lesions depends on the histological and cytopathological findings of endobronchial ultrasound-guided transbronchial needle aspiration, transthoracic biopsy, resected sample of lesions, microbiological examination or clinical follow-up of at least 6 months.ResultsEndobronchial ultrasound multimodal imaging of 97 malignant lung lesions and 19 benign lung lesions from 116 patients were analyzed. There were statistically significant differences in distinct margin, presence of lobulation sign, presence of necrosis, and elastography grading score 4-5 between malignant and benign lung lesions, among which presence of lobulation sign and elastography grading score 4-5 were independent predictors. A diagnostic scoring model was then constructed based on the above four features, and when two or more features were present, the sensitivity, specificity, positive predictive value, negative predictive value and accuracy for malignant lung lesions prediction were 92.78%, 57.89%, 91.84%, 61.11% and 87.07%, respectively.ConclusionsThe combination of endobronchial ultrasound grayscale and elastography has potential value for malignant and benign lung lesions differentiation. The diagnostic scoring model established in this study needs further validation to guide the malignant and benign diagnosis of lung lesions.

Project description:In minimally invasive surgery (MIS), trainees need to learn how to interpret the operative field displayed on the laparoscopic screen. Experts currently guide trainees mainly verbally during laparoscopic procedures. A newly developed telestration system with augmented reality (iSurgeon) allows the instructor to display hand gestures in real-time on the laparoscopic screen in augmented reality to provide visual expert guidance (telestration). This study analysed the effect of telestration guided instructions on gaze behaviour during MIS training. In a randomized-controlled crossover study, 40 MIS naive medical students performed 8 laparoscopic tasks with telestration or with verbal instructions only. Pupil Core eye-tracking glasses were used to capture the instructor's and trainees' gazes. Gaze behaviour measures for tasks 1-7 were gaze latency, gaze convergence and collaborative gaze convergence. Performance measures included the number of errors in tasks 1-7 and trainee's ratings in structured and standardized performance scores in task 8 (ex vivo porcine laparoscopic cholecystectomy). There was a significant improvement 1-7 on gaze latency [F(1,39) = 762.5, p < 0.01, ηp2 = 0.95], gaze convergence [F(1,39) = 482.8, p < 0.01, ηp2 = 0.93] and collaborative gaze convergence [F(1,39) = 408.4, p < 0.01, ηp2 = 0.91] upon instruction with iSurgeon. The number of errors was significantly lower in tasks 1-7 (0.18 ± 0.56 vs. 1.94 ± 1.80, p < 0.01) and the score ratings for laparoscopic cholecystectomy were significantly higher with telestration (global OSATS: 29 ± 2.5 vs. 25 ± 5.5, p < 0.01; task-specific OSATS: 60 ± 3 vs. 50 ± 6, p < 0.01). Telestration with augmented reality successfully improved surgical performance. The trainee's gaze behaviour was improved by reducing the time from instruction to fixation on targets and leading to a higher convergence of the instructor's and the trainee's gazes. Also, the convergence of trainee's gaze and target areas increased with telestration. This confirms augmented reality-based telestration works by means of gaze guidance in MIS and could be used to improve training outcomes.

Dataset Information

Multimodal-GuideNet: Gaze-Probe Bidirectional Guidance in Obstetric Ultrasound Scanning.

Publications

Multimodal-GuideNet: Gaze-Probe Bidirectional Guidance in Obstetric Ultrasound Scanning.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets