RL finally works at scale with LLMs and this market doesn't shoot to 100%? what are we doing

@Bayesian yeah seems like a lock

I feel like GPT 4.5 can play chess better consider it is a larger model than GPT-4o and 4. It plays much better than most LLMs

A serious consideration of this market is whether a Super GM is even going to bother to play a capable LLM by 2028. Im never even half sure that will occur. It would be an interesting novelty for one of the streamers - but who knows which open/free model they'll play against.

@gamedev I think it's a given (95%, say) that if an LLM is reported to be GM level, a super GM will get content out of playing against it. There is a serious risk (and I'm not sure it's intentional*) that they would not play it blind, either by chance or because it is good enough that the GM doesn't feel the need for a handicap.

*There is a conversation to that effect here: https://manifold.markets/MP/will-a-large-language-models-beat-a#YwtaHaw89NHjtTYU5dYC, but it seems against the spirit of the market to resolve No if an LLM is beating GMs and they're not disadvantaging themselves further

@Frogswap my point is that I don't want to have the AI need to be fed the state of the board. But in the year of our lord Jesus Christ of 2025 in which 2.5 Pro spits 1300 lines of python, it seems trivial that it cans haha.

Someone can develop a small benchmark. I bet most SOTA non-reasoning LLMs like Deepseek V3 can deduce the board state from a list of moves. Imagine with reasoning.

3.5 instruct was good because it was evaluated on chess. Now it's not in the benchmarks.

But by 2028 the reasoning models should be search models.

I think it's possible with a fine tuned LLM but the generic LLMs may be grandmaster level. It depends on how much compute they're allowed.

If limited to 4k tokens probably not but if they can do something like Deep Research where they output something on the order of 1M tokens they should be able to reach GM level via searching the game tree.

If Chess isn't a training metric I doubt LLMs can generalize that well without RL (from Stockfish)

Don't want to bet on this because of too much uncertainty on whether anyone puts sufficient effort into incorporating chess abilities.

@ChinmayTheMathGuy I played Sonnet 3.5 against o3-mini-high. Sonnet was playing better most of the early game and gained a piece, got into a better overall position.

Later in the game, both models increasingly lost track of the board, and I had to block more and more illegal moves (sometimes 5 moves before they selected a legal one.) They also generally did not avoid or take advantage of obvious blunders. Sometimes even when their own comment should have revealed the blunder, e.g. "I'll move my queen to c4, attacking your bishop on d5." In fact, this sort of comment usually caused the other model to e.g. retreat the bishop instead of taking the queen. o3 in particular moved a rook deep into Claude's territory, intentionally "attacking" pieces in positions where the rook could be freely taken by a pawn or a bishop multiple times, but Claude never did that until forced to.

Eventually I gave up because the number of illegal moves essentially made the game random.

Would an LLM that implements self training whose dataset includes chess games count?

@DanielSacks idk. We'd need to think about it.

This may be an tougher challenge than expected it has 2 years since 2023 and no LLM has even come close to that elo the only llm that came semi-close in chess by just predicting moves based on its dataset was the mysterious gpt 3.5 turbo instruct. If Llms doesn't start playing chess and does not fail to keep track of the board state. I will have to sell by late 2026 or 2027. This is very concerning since this is almost 2 to 3 years away

@Blocksterpen3 Related, o1 pro lost to me easily (which was only the 2nd game of chess I played in years.) It also repeatedly got confused about the state of the board.

https://chatgpt.com/share/675e2bbb-2e88-8009-8382-b72bd610253c

@DavidBolin yeah I hope o3 or even o4 can play a coherent game of chess. Even deep seek r1 fails around move 13

@DavidBolin

LLMs get better at chess when given three examples of legal moves and their results and asked to repeat the entire previous set of moves before each turn. This can likely be applied to any game.

https://dynomight.net/more-chess/

Is this with no prompting?

Question: if o3 does this, would it resolve as yes? Also what does blind chess mean in context of a language model?

@RossTaylor I'm very confident that o3 will not beat a super GM

@AdamK Doesn’t answer the question - would that resolve as a yes if it did?

@RossTaylor Assuming o3 is also text-only, then yes. The "blind" criterion just means the model doesn't get to see pictures of the board

@AdamK Thanks for clarification! Is model allowed to imagine board states in its chain of thought?

@RossTaylor That's what it would have to be doing implicitly for the CoT to be useful. o3's CoT is almost certainly just text.

It's starting to look like this market is just a countdown to whenever one of the frontier labs decides to apply reasoning post-training to chess.

@AdamK and to whether a super grandmaster is bored enough to play vs an AI at chess.

@AdriaGarrigaAlonso I think that is actually a rather significant component of this question. You could reframe the resolution as "Will a super grandmaster play a serious game of chess against an LLM by 2030?". Even if LLMs continue to improve at chess (they currently aren't any good), this other contingency has to hold as well. Current market seems high.

Has anyone tested o1 or o1-pro on chess? Might be expensive to do, but I feel like it would be interesting. I predict it would not be significantly better than the 1600ish rating of gpt3.5-instruct

@dominic I think the more RLHFed the model is, the worse it is at chess. That's probably why 3.5 instruct is better than 4, 4o, and probably o1.

I might be wrong.

It should be better if the output is constrained to pgn format and fine tuned on stockfish analysis (available on lichess pgn file).

This already a transformer that’s at 2700 just predicting stockfish.

http://arxiv.org/pdf/2402.04494

@dominic It was pretty bad.

https://chatgpt.com/share/675e2bbb-2e88-8009-8382-b72bd610253c