SWE-Lancer evaluates AI agents' ability to complete real-world freelance software engineering tasks sourced from Upwork, mapping performance directly to monetary value.
This market focuses the open-sourced SWE-Lancer Diamond evaluation set, which comprises 502 tasks (237 IC SWE, 265 SWE Manager) collectively valued at $500,800 USD.
The current State-of-the-Art (SotA) reported in the paper for total earnings on SWE-Lancer Diamond is $208,050 USD under a pass@1 metric (achieved by Claude 3.5 Sonnet).

The figures above (taken from the paper) illustrate the key idea behind the benchmark (which spans IC SWE and SWE Manager tasks).
Market Details
Source: This market resolves based on published data from the maintainers of the SWE-Lancer benchmark (e.g., OpenAI researchers listed in the paper or designated successors) or credible third-party evaluations using the official benchmark configuration and SWE-Lancer Diamond set.
Metric: Total Payout Earned (USD) on the SWE-Lancer Diamond set. This is the sum of the real-world payouts associated with each successfully completed task (pass@1) in the Diamond set (across both IC SWE and SWE Manager tasks).
Target Score: Greater than or equal to $400,000.00 USD.
Reporting Window: The score must be achieved by an AI agent and credibly reported (e.g., in a peer-reviewed publication, arXiv preprint, official leaderboard, major AI lab report, public repo) before December 31st, 2025, 23:59 UTC.
Resolution Criterion
This market resolves to YES if the State-of-the-Art (SotA) Total Payout Earned on the SWE-Lancer Diamond set is credibly reported to have reached or surpassed $400,000.00 USD within the reporting window.
Otherwise, the market resolves to NO.
Market Closing Date
The market will close on January 15, 2026, 00:00 UTC, to allow for potential reporting delays. It may resolve earlier if the YES condition (>= $400,000.00 USD reported) is met and confirmed before this date.