{"database":"biostudies-literature","file_versions":[],"scores":null,"additional":{"submitter":["Maier-Hein L"],"funding":["NIBIB NIH HHS","Swiss National Science Foundation","NINDS NIH HHS","Wellcome Trust"],"pagination":["5217"],"full_dataset_link":["https://www.ebi.ac.uk/biostudies/studies/S-EPMC6284017"],"repository":["biostudies-literature"],"omics_type":["Unknown"],"volume":["9(1)"],"pubmed_abstract":["International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observer"],"journal":["Nature communications"],"pubmed_title":["Why rankings of biomedical image analysis competitions should be interpreted with care."],"pmcid":["PMC6284017"],"funding_grant_id":["205321","157207","R01 NS070906","R01 EB017230"],"pubmed_authors":["Feldmann C","van der Sommen F","Honauer K","van Ginneken B","Eisenmann M","Rajpoot N","Taha AA","Kopp-Schneider A","Menze BH","Wang CW","Reinke A","Onogur S","Neher PF","Marz K","Stankovic M","Bogunovic H","Speidel S","Zheng G","Arbel T","Frangi AF","Scholz P","Landman BA","Niessen W","Maier-Hein L","Maier-Hein K","Weber MA","Jannin P","Maier O","Stoyanov D","Carass A","Hanbury A","Full PM","Kozubek M","Bradley AP","Sirinukunwattana K","Sharp GC","Stock C","Muller H"],"additional_accession":[]},"is_claimable":false,"name":"Why rankings of biomedical image analysis competitions should be interpreted with care.","description":"International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observer","dates":{"release":"2018-01-01T00:00:00Z","publication":"2018 Dec","modification":"2025-04-26T02:58:28.415Z","creation":"2019-03-26T22:43:38Z"},"accession":"S-EPMC6284017","cross_references":{"pubmed":["30523263"],"doi":["10.1038/s41467-018-07619-7"]}}