By when will 85% be reached on the public evaluation set on ARC-AGI-Pub?
💎
Premium
65
á¹€56k
2030
91%
01/01/2026
92%
01/01/2028
95%
01/01/2030
Resolved
NO
01/01/2025

See https://arcprize.org/arc-agi-pub.

This allows for arbitrary API usage unlike the ARC-AGI prize.

I will resolve based on when the date when the method is first reported even if it is not submitted until substantially later (due to e.g. no public API).

I will apply a restriction roughly in line with the $10,000 runtime cost contraint, though it is unclear how this should resolve with respect to private models with no known API price.

I will exclude solutions that seem heavily overfit based on the semi-private evaluation set.

(Cut off data is resolved based on midnight PT.)

#AI
#Technical AI Timelines
Get á¹€1,000 play money
4 Comments
Sort by:
01/01/2025

@RyanGreenblatt I think this resolves NO? o3 did pass 85%, but it was well above the cost constraints.

bought á¹€45 01/01/2025 YES

https://arcprize.org/blog/oai-o3-pub-breakthrough

sold á¹€477 01/01/2025 YES

wow, 83% on public with high efficiency. So close (and I feel the mana punishment for trading too fast before reading the fine print)

If 85% is reached on the main ARC-AGI prize track and there isn't a clear reason to think this wouldn't transfer to ARC-AGI-Pub, I would also resolve this market to true even if no one submits to ARC-AGI-Pub.

Related questions