{"database":"biostudies-literature","file_versions":[],"scores":null,"additional":{"omics_type":["Unknown"],"volume":["649(8099)"],"submitter":["Center for AI Safety"],"pubmed_abstract":["Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding<sup>1</sup>, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer"],"journal":["Nature"],"pagination":["1139-1146"],"full_dataset_link":["https://www.ebi.ac.uk/biostudies/studies/S-EPMC12851929"],"repository":["biostudies-literature"],"pubmed_title":["A benchmark of expert-level academic questions to assess AI capabilities."],"pmcid":["PMC12851929"],"pubmed_authors":["Kratish Y","Kazibwe Z","Douville G","Rodrguez RAH","Men Q","Alley W","Dias FM","Pantidis I","Bashmal L","Klose I","Rank B","Christof B","Mollaei S","Sperzel M","Sidhu J","Yuan Q","Shahrtash H","Kean J","Yuan J","McInnis IMJ","Jagota M","Somrak M","Avadhanam A","Shumar O","Amaro B","Bailey J","Berger Z","Sparrow C","Yu M","Cao H","Zhu Z","Yu S","Qi H","McAlister IC","Tawfeek AR","Qi T","McCarty B","Yu E","Shenoy A","Zhou YA","Yoon J","Maksapetyan A","Rossbach V","Sayed M","Scotese CR","Zekry M","White C","Yu Z","Rawal K","Wei B","Wei A","Gonzalez DE","Hernandez-Orallo J","Mishra R","Mishra S","Semler A","Purohit SS","Hava DB","Bari MS","Riblet R","Scale AI","Gulati S","Mikov A","Albanie S","Abramovitch M","Stade J","Zhu K","Poswiata R","Zang A","Dong W","P R","Agarwal KP","Richmond M","Ivanov S","Terpin A","Ragoler A","Rao A","Xiong Y","Sharma A","Bian S","Ceconello C","Agarwal A","Chen Z","Of Arc Xavier J","Chen Y","Hoover A","Chen S","Leventov R","Goldfarb A","Ivanov A","Gundlach H","Chen Q","Lu XH","Bharath V","Chen K","Plassart A","Chen J","Chen M","Twayana M","Montecillo RG","Harjadi C","Wehr D","Hoback S","Rosset S","Chen H","Chen B","Laurendeau J","Plecnik R","Gonzalez JC","Chen E","Martinez D","Goryachev V","Galal A","Sakor A","Shiyu H","Nguyen TH","Ma Z","Ma W","Nguyen TA","Fournier-Facio F","Newbould J","Johnson B","Hendrycks D","Racz B","Votava J","Maximiano G","Tarver T","Bogdanik S","Dernoncourt F","Phillips S","Maghsoudimehrabani M","Pouriamanesh R","Bugas D","Ma B","Anderson E","Poulos J","Anderson D","Patramanis D","Sala S","Preu T","Willcocks CG","Shah N","Shen J","Brenner MP","Shah S","Jindel K","Shah J","Finke L","Shen Z","Vincze M","Shah T","Salazar J","Karwowski J","Dehghan A","Bateman G","Pu B","Halevy K","Feng SY","Mahmood M","Reidegeld F","Peterson M","Maung E","Dworakowska B","Wecker A","Dhuliawala S","Gehrunger T","Haggstrom O","Lucki J","Venugopalan S","Han X","Han Z","Quinn Liu H","Cole J","Vieira BH","Oller M","Meril E","Nayak A","Phan L","Lindsey J","Fosin I","Farhidi F","Sayous R","Apronti A","Saha A","Plomecka M","Scipio EJ","Bartlett CW","Pflugfelder R","Yap L","Mariji H","Schneider K","Gerbicz R","Hallman E","Agrawal A","Brown EDL","Kolt N","Perlitz Y","Clarke D","Peng C","Tran H","Agrawal R","Lang L","Patlan AS","Zubic N","Nunes MIS","Iyer K","Aly R","Mendoza S","Sarti G","Santos TCB","Francois P","Toukmaji C","Hernandez-Camara P","Enyekwe I","Singla V","Vinnikov V","Antonenko DS","Manini D","Kretov E","Weng E","Kansal Y","Eicher J","Nguyen MC","Battaglia M","Bizeul A","Abdoli S","Hoyer D","Marion P","Amit A","Gimenez J","Moat R","Dhingra S","Morak W","Patwardhan T","Choi M","Zhidkovskaya AB","Gul S","Krause M","Olszewska K","Cho N","Gendron G","Goswami LP","Yan M","Cheatom A","Mohamed M","Stanley R","Shen A","Chopra A","Gitter A","Matsumoto K","Coppola BP","Albani G","Khan S","Makarychev Y","Lin X","Habibi F","Jain RK","Datta S","Zohar O","Lamparth M","Khoja A","Yalam S","Banik D","Heilala V","Taylor E","Demian C","Sauers S","Qazi U","Stump C","Wang HK","Rannev I","He H","He M","Bodur OF","Stritecky V","Bezzi D","Gloor L","Jain E","Ebert S","Goncalves DV","Ponkshe K","Mavroudis V","Gonzalez J","Safdari M","Pai K","Lin H","Jain Y","Tucker-Foltz J","Colino J","Adesanya F","Chidambaram M","Yong ZX","Fruhauff T","Malishev D","Jones A","Medley G","Wist J","Mensah S","Fan H","Maayan G","Fan J","Karakoc A","Li B","Stambaugh N","Kim DH","Sandlin J","Ai V","Wang JP","Piperski A","Padlewski P","Spelda P","Fan Y","Vaquero L","De Maddalena R","Nevirkovets V","Hollom L","Chalstrey E","Kassahun H","Geirhos R","Zheltonozhskii E","Lewark L","Schut L","Yifei LS","Tordera D","Segev B","Turchin P","Gupta H","Akov I","Gupta I","Zhelnov P","Stander D","Geirhos C","Cunningham DM","Li H","Li J","Li C","Reddy SP","Li D","Looi SZ","Li F","Li R","Li L","Peristyy A","Tu H","Li N","Stokes AC","Li X","Li Y","Rodriguez-Romero A","Li S","Magoulas I","Saxena P","Schottdorf M","Le VK","Ducey J","Dhole K","Mao M","Grover R","Halasyamani S","Zebaze AR","Lengler J","Ramirez-Trinidad A","Friedrich A","Jiang T","Cozianu C","Shabani A","Jiang M","Jiang Y","Jackson A","Jiang X","Morris J","Nam J","Zheng G","Stendall R","Zheng E","Jia Z","Verma S","Yin Y","Brianski M","Malysheva Y","Shi JWL","Siddh S","Maksimov AP","Luo G","Malina S","Luo E","Burns N","Zhang W","Raynor B","Lauff R","Askew L","Givre A","Zhang T","Noye M","Luo J","Heimonen JP","Zhang Z","Zhang Y","Zhang X","Yin M","Zhang G","Telluri A","Zhang D","Paradise O","Narayan H","Sunkari S","Zhang H","Moulin A","Zhang O","Bouyamourn A","Zvonkine D","Zhang M","Zhang L","Bosca MC","Shinde DP","Zhang S","Zhang R","Moyano AJ","Zhang Q","De Luca GB","Jin A","Sponselee K","Jin C","Crisostomi D","Ni C","Jin J","Zhang A","Ha J","Petruzella G","Kamdoum AGK","Elkhanany A","Rozhon V","Senthilkumar S","Mukherjee K","Hauser J","Yucel AT","McGowan J","Clark R","Rissone P","Arnaboldi L","Slen A","Roth M","Hoerr RG","Borbas B","Santens T","Abbondanza D","Del Rio-Chanona M","Jeanplong F","Zhou A","Chu E","Malusare A","Le A","Zhou K","Galgon G","Ferreira D","Xiao Y","Dodonov D","Kelley E","Soroko I","Xiao T","Nasim Z","Zhou Y","Zhou X","Haffenden H","Batra H","Pottmaier D","Mundler N","Luczyna G","Karim L","Brown A","Carroll M","Sun C","Schoelkopf H","Pronin A","Fraga NW","Andreoletti J","Gopi S","Kumar P","Kazakov D","Noever D","Giordano P","Sun R","Sun D","Moller S","Sun G","Wykowski J","Golshani B","Lauer P","Ferret J","Mazeika M","Gustafsson N","Sun K","Subramanian S","Yang H","Le L","Yang J","Drori J","Tan J","Kang T","Sun T","Brooks M","Radhakrishnan A","Huber LS","Taheri O","Sun X","Sun Y","Sun Z","Yang X","Labrador YCL","Shaaban M","Yang R","Yang T","Usawasutsakorn S","Sundarapandiyan V","Sanker V","Bacho A","Mak J","Zweiger A","de Oliveira Santos E","Koppel J","Press O","Bacho K","Uzhou M","Delaney E","Bacho R","ElSheikh A","Tee A","Robinson J","Hausenloy J","Kassani P","Shukla A","Brown Z","Winter N","Demircali AA","Yacar L","Demircali AE","Baranov A","Kumar H","Mourrat JC","Aji AF","Kazakov M","Shukla P","Bigi F","de la Rosa F","Cavalleri S","Shah IA","Kharlamova A","Sztyber-Betley A","Loiseau G","Overholt KJ","Duersch J","Center for AI Safety","Manik T","Balabanian G","Barkan S","Chen PE","Aldous D","Van Der Sypt L","Fossum CJ","Sims B","Xu R","Creighton S","Xu H","Xu J","Souza V","Carmi A","Recchia G","Xu S","Scaramuzza D","Kvistad L","Xu Z","Bohdal O","Kopylov A","Ye Z","Pyda D","Riis S","Imperial JM","Stehberger M","Brussel L","McKellips G","Meiburg A","Gangal V","Wydallis JB","Li WD","Mostaghimi H","Leser JN","Doroshenko M","Korbak T","Mirabadi AK","Yilmaz A","Yeadon W","Muzhen J","Heydari N","Siedler PD","Du Z","Chern S","Bar K","Eron M","Salauddin SM","Marjanovic SV","Dick AK","Popescu M","Barzowski S","Green SR","Kulkarni V","Ambay JA","Islam M","Steinerberger S","Bradshaw P","Lai J","Liu J","Liu M","Sewuster G","Leung J","Vargus F","Lopez-Miguel ID","Obikoya GD","Poesia Reis E Silva G","Liu Y","Liu Z","Lukas M","El-Wasif Z","Sonmez Y","Nandor M","Sakal T","Cojoc D","Brito-Santana L","Guan J","Poritski V","Dobarskyi K","Tire K","Gatti A","Pokutnyi O","Munro D","Pandey R","Adoul Z","Elser V","Loume GA","Menshawy A","Knights T","Harris C","Crowson SJ","Lai S","Hammon A","Petersen P","Zu Y","Platnick J","Hulbert D","Kalinin M","Shulga N","Liang Y","Todoran S","Hao W","Yaln Y","Arrais AO","Hoang H","Pondaven A","Balint B","Liang K","Ponti EM","Malod G","Bosio C","Verkama E","Crozier J","Chhablani G","Vergo E","Basler L","Robins J","Gusev I","Quod Soler Bartomeu D","Thi Hoang H","Bosio I","Cartwright A","Singer E","Stepanic S","Weller O","Radionov M","Rodrigo-Gines FJ","Zhang CBC","Kazemi F","Ren Y","Zsambok J","Uro JL","Shepherd M","Tanwie NM","Naiya MM","Ermon S","Jackson JW","Tiryakioglu M","Qu X","Ren R","Ren Q","Parcalabescu L","Dewerpe I","Xin L","Montgomery K","Dendane A","Ovchynnikov M","Singh A","Ling J","Chang Y","Wong AAWL","Sanchez A","Pang TC","Fu E","Jansen T","Suresh S","Costa M","Vajipey V","Bogdanov S","Roberts J","Perelkiewicz M","Angquist I","Kamalov F","Sommerhage F","Maheshwari R","Siddiqi MR","Iskra O","Kirtland A","Ge J","Hossain Z","Singh M","Michael Pokorny R","Hausknecht K","Shamseldeen S","Sen A","Ze-An N","Singh V","Yao J","Zenitani K","Rogers TR","Piza M","Gurdogan H","Popov S","Arsene P","Zampese J","Manolescu C","Vaz WS","Milliron H","Thaman K","Ekstrom F","Feng F","Ginis V","Juefei-Xu F","Yakimchyk A","Fasiludeen AA","Kasamsetty R","Dieuleveut A","Wang EY","Shridhar K","Tony Lian L","Krishna S","Jaeger J","Fish S","Faraboschi P","Merrill W","Vilchis VEG","Chung JW","Bran AM","De Oliveira Pena R","Perrella D","Resman N","Finocchio R","Arthornthurasuk S","Vilchis TG","Pan J","Ali S","Yuan MX","Pan R","Matos JO","Gross J","Grabb D","Perry S","Szlyk H","Patel A","Remy N","Fang J","Ardito CG","Hu Y","Nagumalli L","Hu W","Deng J","Deng M","Chigurupati S","Algaba A","Durand S","Patel O","Pradeep R","Patel S","Maggetti M","Mofayezi M","Lynch J","Kostakos V","Liakhovitskaia A","Menezes A","Stolfo A","Thornley E","Daans N","Ramakrishnan K","Prabhu A","Reuel A","Sanz-Ros J","Hu J","Do L","Goshu GM","Martinson S","Mohammadzadeh F","Rodriguez MO","Zouhar V","Motwani S","Zhao X","Zhao S","Dhamane O","Zhao H","Ciobaca S","Caetano DM","Zhao B","Cohen MK","Cohen N","de Witt CS","Jha K","Kuchkin A","Cheraku A","Meer K","Sahu A","Kirchhof M","Kuchkin V","Elgnainy H","Beger C","Pageler B","Loader J","Avagian D","Alarab I","Duc E","de Oliveira Junior BA","Kaddour J","Burdzy K","Aziz MF","Berkani M","Vetoshkin L","Pereira R","Singhal R","Wolff H","Agu C","Nattanmai A","Scaria KJ","Cavanagh JM","Giska F","Lass J","Firsching M","Lux TCH","Stap D","Reddish J","Mehkary M","Di Fratta C","Cosma A","Verbeken B","Aggarwal D","Outevsky D","Kondor D","Muennighoff N","Clavier P","Medved T","Cai M","Cai J","Sivarajan S","Zou A","Orel D","Zou C","Xi H","Echeazu D","Held W","Guillod J","Khanh TQ","Okutsu K","Rodola E","Kalpathi T","Paek J","Lackner M","Weber B","Lim D","Kreuer J","Weber S","Alapont X","Oliveira E","Soori S","Cai W","Kreiman T","Shi S","Taamazyan V","Aaron A","Shapourian H","Shi T","Portier J","Milbauer J","Villanueva JP","Martin F","Cao PM","Jallon A","Zhuang C","Friday C","Osbey T","Coleman S","Pham DT","Bartolo M","Farina N","Maar J","Perez RH","Kaddar Y","Park HK","HLE Contributors Consortium","Heidinger H","Zhang DK","Yeo W","Levin JC","Van den Houte K","Chandok N","Caciolai A","Efremov D","Vendrow E","Agnihotri S","Myklebust B","Vendrow J","Piccardo M","Chernyavsky J","Lee S","Chernyavsky I","Franca A","Anugraha D","Zhang CE","Kim H","Ma JJ","Kim E","Qin J","Lee J","Tang C","Demir G","Peskoff D","Dai Y","Meding K","Cheng Z","Tang H","Tang M","Cheng V","Tang L","Tang N","Garretson AC","Wheeler R","Goertzen T","Kaparov N","Hebbar SA","Kamphuis T","Briia G","Tam J","Andrew Favre DO","Zsolnai-Feher K","Hartman TD","Sherman G","Rabern B","Tkadlec J","Ben-David S","Kim S","Kim R","Dai L","Kim W","Qin H","Lee Y","Rodman E","Redenti A","Lee KH","Yue S","Wu J","Sinha A","Ler D","Wang A","Samal S","Wang C","Patil D","Yue R","Wang D","Wu B","Wang G","Lo EJY","Wang J","Wang K","Wang M","Patil V","Foster M","Hogg T","Kirchner JH","Minissi AMR","Wang S","Wang T","Sucholutsky I","Achilleos A","Wu S","Holmes D","Sridhar K","Wu T","Nhu AN","Wang X","Samuele F","Wang Y","Wu W","Wang Z","Huy TD","Park H","Park I","Park J","Duclosel D","Handoko","Park N","Mahfoud M","Ho A","Inlow MH","Xian R","Smucker L","Ho L","Pinto A","Anantheswaran U","Loh KY","Pinto F","Wu A","Whitfill P","Veith J","Penaflor M","Arkhipov P","Sun HY","Lorena R","Suhail A","Chamorro-Padial J","Park CF","Giboney Z","Khoury S","Ansarinejad B","Rosu P","Williamson D","Krenek K","Gritsevskiy A","Rismanchian S","Song Z","Nie Y","Joshi P","Carauleanu M","Utpala S","Nguyen T","Song D","Tomasiello A","Campbell R","Kon PTJ","Nguyen J","Degorre J","Dasouqi A","Nguyen L","Winata GI","Schwaller P","Nguyen H","Schmitt J","Crispino N","Sileo D","Shahid SM","Ma MQ","Bahaloohoreh M"],"additional_accession":[]},"is_claimable":false,"name":"A benchmark of expert-level academic questions to assess AI capabilities.","description":"Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding<sup>1</sup>, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer","dates":{"release":"2026-01-01T00:00:00Z","publication":"2026 Jan","modification":"2026-06-12T10:17:43.814Z","creation":"2026-06-12T03:12:36.223Z"},"accession":"S-EPMC12851929","cross_references":{"pubmed":["41606155"],"doi":["10.1038/s41586-025-09962-4"]}}