When will SOTA on MMLU STEM be >= 90%? | Manifold

When will SOTA on MMLU STEM be >= 90%?

Mini

7

Ṁ374

2032

1D

1W

1M

ALL

4%

Before 2023

8%

2023-2024

36%

2024-2025

24%

2025-2026

14%

2026-2027

4%

2027-2032

8%

2032-never

MMLU: https://arxiv.org/abs/2009.03300v3.

#Technical AI Timelines

Get Ṁ1,000 play money

Sort by:

GPT-4 is 82.7% https://arxiv.org/pdf/2303.12712.pdf#page=36

Metaculus says 91 in mid-2025, STEM is one of the hardest. I think it's unlikely to happen before 2025.

What do these answers mean? If it happens in 2025, will 2024-2025 and 2025-2026 both be chosen?

@NcyRocks Sorry: python style ranges, so X - Y means after the start of X and before the start of Y. If I can I'll edit the options to make that clearer

Related questions

MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?

MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?

Will SOTA on MATH in Sep 2024 utilize a hard-coded search/amplification procedure?

When will no-calculator SOTA on the MATH dataset pass 90%?

BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?

MMLU 99% #3: Will SOTA for MMLU (average) pass 99% by the start of 2026?

MMLU 99% #2: Will SOTA for MMLU (average) pass 99% by the start of 2025?

By 2026, will it be standard practice to sandbox SOTA LLMs?

Will there be a period of 12 contiguous months during which no new compute-SOTA LM is released, by Jan 1, 2033?

Will the transformer architecture be replaced in SOTA LLMs by 2028?

Related questions

MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?

MMLU 99% #3: Will SOTA for MMLU (average) pass 99% by the start of 2026?

MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?

MMLU 99% #2: Will SOTA for MMLU (average) pass 99% by the start of 2025?

Will SOTA on MATH in Sep 2024 utilize a hard-coded search/amplification procedure?

By 2026, will it be standard practice to sandbox SOTA LLMs?

When will no-calculator SOTA on the MATH dataset pass 90%?

Will there be a period of 12 contiguous months during which no new compute-SOTA LM is released, by Jan 1, 2033?

BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?

Will the transformer architecture be replaced in SOTA LLMs by 2028?