Will an open source model beat GPT-4 in 2024?
66
Ṁ22k
Dec 31
49%
chance

Since the release of GPT-4 by OpenAI, the AI field has witnessed rapid advancements. Open source models, leveraging collaborative and transparent development, have been a significant part of this innovation. This question seeks to forecast whether an open source model will surpass GPT-4(-1106) in terms of capabilities by the end of 2024.

Criteria for Resolution:

  1. Performance Metrics: The open source model must demonstrate superior performance compared to GPT-4(-1106) in at least X standardized major benchmarks. [Will specify later]

  2. Public Acknowledgment: The superiority of the open source model over GPT-4 must be recognized by a consensus in the AI research community, evidenced through published papers, independent benchmarking reports, or notable industry acknowledgments.

  3. Open Source Verification: The model must be verifiably open source, with its codebase freely available to the public.

#AI
#OpenAI
#LLMs
#GPT-4 speculation
Get Ṁ1,000 play money
12 Comments
Sort by:

The answer here is almost certainly going to be "yes if llama counts as open source" (meta calls it open source, but eg Simon Willison would insist it's only "open weights")

The resolution criteria pretty clearly says “with its code base open to the public.” I tried to get an answer to this below but OP appears to be inactive

so that'd mean this would resolve NO? why not bet it down using this line of reasoning

bought Ṁ250 NO

ig you already have a large % of your nw on NO so it's reasonable not to put more. Ig this is a pretty strong case for NO since afaict Llama 3.1 405B doesn't have its codebase freely available to the public, so doesn't satisfy the resolution criteria. tbc I'm not speaking in my capacity as a mod, just confused about the market price

opened a Ṁ2,000 NO at 35% order

Just hedging against the mods interpreting things differently but yeah agreed my odds of that were lower than 60

Would Llama 3 be considered open source under the resolution criteria?

Looking good

predicts YES

"The model must be verifiably open source" - by "open source", do you mean just "available", or do you mean this (widely-accepted) definition: https://opensource.org/osd/

"The model must be verifiably open source, with its codebase freely available to the public." -- should also the model weights (parameters) be freely available? I recommend yes.

What does "its codebase freely available to the public" mean? Does a torrent containing just the weights (a la Mistral) count?

@dp I recommend it doesn't count, since from the weights alone, the model cannot be independently executed

Related questions