When will a single model first achieve 10@k solve rate >= 90% on the CodeContests dataset? | Manifold

When will a single model first achieve 10@k solve rate >= 90% on the CodeContests dataset?

Mini

7

Ṁ306

2032

1D

1W

1M

ALL

4%

Other

5%

Before 2023

11%

2023-2023

17%

2024-2025

25%

2025-2026

18%

2026-2027

17%

2027-2032

1.6%

2032-never

10@k - unlimited samples may be generated, but at most 10 code samples may be submitted per problem. I will post my own responses and pick whichever one of those turns out to be correct. CodeContests dataset: https://github.com/deepmind/code_contests Close date updated to 2050-01-01 11:59 pm Apr 8, 4:16pm: Options are written as [a, b), so 2024-2025 means occurring after the start of 2024 and before the start of 2025. Close date updated to 2032-01-01 11:59 pm

#Technical AI Timelines

Get Ṁ1,000 play money

Sort by:

Currently, it's only 30%. Definitely, 90%+ is not going to be reached this year. Extremely unlikely (1% maybe?) next year too.

@ValeryCherepanov Yeah the before 2023 option seems like free money to me, I think the market is just old and people don't have it on their radar.

Ugh, sorry everyone, this should read 2023-2024, as in "occurring in 2023"

Related questions

Will any model get above human level (92%) on the Simple Bench benchmark before September 1st, 2025.

Will the winner of the ARC Kaggle competition solve more than 50% of the tasks

AI resolves at least X% on SWE-bench without any assistance, by 2028?

Will I get 2100+ on codeforces in 2024?

Will any AI be able to formalize >=90% of IMO problems by the start of 2025?

When will an AI model be better than me at competitive programming?

By when will SAT solvers be 10x faster (than 2023 on same hardware)?

When will AI be able to solve 50% of official Jane Street puzzles?

Will an AI achieve a perfect score on the Miklós Schweitzer Competition before 2035?

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?

Related questions

Will any model get above human level (92%) on the Simple Bench benchmark before September 1st, 2025.

When will an AI model be better than me at competitive programming?

Will the winner of the ARC Kaggle competition solve more than 50% of the tasks

By when will SAT solvers be 10x faster (than 2023 on same hardware)?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

When will AI be able to solve 50% of official Jane Street puzzles?

Will I get 2100+ on codeforces in 2024?

Will an AI achieve a perfect score on the Miklós Schweitzer Competition before 2035?

Will any AI be able to formalize >=90% of IMO problems by the start of 2025?

Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?