Currently this market has an expected average score of 53.6 which I think is quite high. Especially given how the neural scaling laws seem to be coming home to roost.

lots of arb possible with my market https://manifold.markets/jim/when-will-humanitys-last-exam-be-sa

Are models using tools and/or performing web search eligible under current resolution criteria?

@Metastable I think no matter what tools you add to AI, it still remains AI

@mathvc ability to livechat with human experts?

@jim with an exception of using humans 🙂, then it’s definitely not artificial intelligence

@mathvc live access to web is maybe somewhere on that spectrum tho

@jim why you call it live access? It doesn’t go to math forum and make a post about math problems.

You can replicate internet access by scraping it and using as a giant database

@mathvc yeah but using giant database or the web seems like it's less reliant on the AI model's innate knowledge and intelligence, more reliant on human knowledge and intelligence.

i've edited the market description a bit to not be dependent on my own discretion for what model counts or doesn't count. now it uses a consensus of reliable sources or moderator consensus, instead of my own opinion. 🤷‍♂️ probably won't come up anyway but i realized i was amassing a decent position so

Anything 30+ is honestly scary territory. FrontierMath is really impressive and all, and no doubt surprising, but kinda an oh that happened type thing. This is the kind of test that would make me seriously reconsider my beliefs about AGI, great market!

ooo maybe a market on which of those two is solved at 80% or above first

@Bayesian sounds like an incentive to finetune my deepseek-giga-overfitter-hle-memorized-v1 model by EOY

@Ziddletwix yeah but that would be CHEATING! and the leaderboard thing would CATCH IT

@Bayesian most likely, but maybe they'll just put an asterisk and scold it in a footnote for being sus & bad. unclear how enforcement is actually handled in practice

fkkkk they might put the footnote saying it's sus affffff then what are we gonna do

@copiumarc I don’t think HLE is harder than Frontier Math.

/Bayesian/which-of-frontiermath-and-humanitys

@mathvc @copiumarc may the person with the best model of reality win

Surely o3 will get >20%?

@qumeric if the benchmark is knowledge heavy it might not do that much better than 4o? prolly will tho. just some low chance that it doesn't

“The dataset consists of 3,000 challenging questions across over a hundred subjects. We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting.”

Well sorry but people are gonna overfit to this. Who is gonna judge whether the model is overfitted or not?

@mathvc yes i am confused by this point. so if some model near EOY is massively overfit to HLE, scores 90%+, and they chime in "yeah its performance wasn't so crazy strong on our few holdout problems, it probably overfit a bit", that still counts as 90%+ right? is the holdout set just used as a separate confirmation of overfitting, its not incorporated to the main score?

i agree this is troubling. What do you think would be the best way to proceed?

@Bayesian i found that scale.ai and safe.ai partnered to create this benchmark and it seems that they keep up-to-date evaluations of all frontier models:

https://scale.com/leaderboard/humanitys_last_exam

I guess we can trust their judgment? That is, they will not put a clearly overfitted model on the leaderboard since it makes the leaderboard useless

@Bayesian i dunno i think all benchmarks have caveats so i'd just pick some source for what each model has achieved on the benchmark & if their screener for overfitting is weak that's kinda priced in

@mathvc yeah i agree, that probably works well enough. will add to the description