When will SWE-bench be solved?
Mini
5
Ṁ28
2030
7%
2024
32%
2025
26%
2026
7%
2027
7%
2028
7%
2029
15%
later

Resolves to the first year in which https://www.swebench.com posts a 90% resolved score in the “verified” category. The current score is 38.80% by Amazon Q Developer Agent.

Get Ṁ1,000 play money
Sort by:
bought Ṁ5 later YES

Relatively significant chance of never (which resolves at later). Frontier LLMs are just going to get more and more contaminated with this benchmark's test data - at some point authors might just stop tracking