Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?
14
Ṁ350
Jan 1
62%
chance

In this tweet (https://twitter.com/ajeya_cotra/status/1684358475416064001?s=20), Ajeya Cotra (admirably) predicted that there's >50% chance >50% of the tasks in the newly announced WebArena benchmark will be solved by a single agent. Note that Ajeya didn't specify that a single agent had to solve all of them but I will resolve based on that, so there is the possibility of divergence.

#AI
#Technical AI Timelines
Get Ṁ1,000 play money
3 Comments
Sort by:

Any reason this (blog post) shouldn't qualify to resolve to "Yes"?

The official WebArena leaderboard also now shows Jace with a >50% result.

For a baseline of current status: the paper author's tweet thread

Completing such realistic tasks is challenging. Our best GPT-4 agent achieves a limited end-to-end task success rate of 10.59%

Understanding HTML with Large Language Models provides some evidence that bidirectional encoder-decoder models outperform GPTs on understanding raw web page HTML, but this benchmark includes more than that:

  1. raw web page html

  2. pixel-based screenshot

  3. accessibility tree of the webpage. Seems like this is a subset of the html DOM tree

Related questions