biostudies-literatureUnknown15(1)Gao YMulti-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited Nature communications9613https://www.ebi.ac.uk/biostudies/studies/S-EPMC11544255biostudies-literatureAn explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer.PMC11544255Kok MHan LZhou HYHe MMann RLongo VWeir Avan Duijnhoven FHLi XLiu ZVentura-Diaz SZhang THorlings HMGao YTan TWang XBeets-Tan RTeuwen JXu ZD'Angelo AfalseAn explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer.Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited 2024-01-01T00:00:00Z2024 Nov2026-07-16T19:52:15.496Z2025-04-06T02:51:45.381ZS-EPMC115442553951114310.1038/s41467-024-53450-8