biostudies-literatureDorent RWellcome Trust448-458https://www.ebi.ac.uk/biostudies/studies/S-EPMC7615858biostudies-literatureUnknown2023We introduce MHVAE, a deep hierarchical variational autoencoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available.Medical image computing and computer-assisted intervention : MICCAI ... International Conference on Medical Image Computing and Computer-Assisted InterventionUnified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations.PMC7615858203148Kapur TJoutard SKogl FGolby AVercauteren TDorent RJuvekar POurselin SFrisken SWells WMHaouchine NTorio EfalseUnified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations.We introduce MHVAE, a deep hierarchical variational autoencoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available.2023-01-01T00:00:00Z2023 Oct2026-06-01T20:15:05.679Z2026-05-20T03:08:35.617ZS-EPMC76158583865538310.1007/978-3-031-43999-5_43