biostudies-literatureUnknownSaha USNIGMS NIH HHSRecent advances in machine learning methods for materials science have significantly enhanced accurate predictions of the properties of novel materials. Here, we explore whether these advances can be adapted to drug discovery by addressing the problem of prospective validation - the assessment of the performance of a method on out-of-distribution data. First, we tested whether k-fold n-step forward cross-validation could improve the accuracy of out-of-distribution small molecule bioactivity predictions. We found that it is more helpful than conventional random split cross-validation in describing the accuracy of a model in real-world drug discovery settings. We also analyzed discovery yield and novelty error, finding that these two metrics provide an understanding of the applicability domabioRxiv : the preprint server for biology2024.07.02.601740https://www.ebi.ac.uk/biostudies/studies/S-EPMC11245006biostudies-literatureStep Forward Cross Validation for Bioactivity Prediction: Out of Distribution Validation in Drug Discovery.PMC11245006R35 GM122547Saha USVendruscolo MCarpenter AESingh SBender ASeal SfalseStep Forward Cross Validation for Bioactivity Prediction: Out of Distribution Validation in Drug Discovery.Recent advances in machine learning methods for materials science have significantly enhanced accurate predictions of the properties of novel materials. Here, we explore whether these advances can be adapted to drug discovery by addressing the problem of prospective validation - the assessment of the performance of a method on out-of-distribution data. First, we tested whether k-fold n-step forward cross-validation could improve the accuracy of out-of-distribution small molecule bioactivity predictions. We found that it is more helpful than conventional random split cross-validation in describing the accuracy of a model in real-world drug discovery settings. We also analyzed discovery yield and novelty error, finding that these two metrics provide an understanding of the applicability doma2024-01-01T00:00:00Z2024 Jul2026-05-01T03:22:27.723Z2025-04-04T15:01:41.923ZS-EPMC112450063900540410.1101/2024.07.02.601740