Presenting the SWE-Lancer benchmark

Can Frontier AI Models Really Earn $1 Million Doing Freelance Coding Work?

A new benchmark called Remote Labor Index puts large language models to the test in a surprising way: instead of scoring them on abstract puzzles, it drops them into real freelance jobs from an actual marketplace and asks whether they could have delivered the work — and gotten paid.

If you follow AI progress, you've probably seen model after model post dazzling scores on coding benchmarks. But those tests usually involve clean, self-contained problems with known answers. RLI takes a different route. It gathers genuine project postings from Upwork-style platforms — the kind of gigs human freelancers compete for — and measures how well AI systems handle them end to end.

What the benchmark actually does

Each task in the set is a real job listing, complete with the client's requirements, any attached files, and the budget the job actually paid. The model gets the same brief a human freelancer would receive, produces a deliverable, and then its output is judged against what the client actually accepted. Payment only counts if the work meets the bar.

The headline question the researchers posed is delightfully concrete: if you hired an AI agent instead of a human, could it accumulate $1 million in completed gigs? Think of it as converting benchmark points into dollars.

How the models performed

The results were humbling for the AI side. The strongest frontier models — including systems like GPT-5 and Claude Sonnet 4.5 — managed to complete only a modest fraction of the jobs they attempted, translating to earnings in the thousands of dollars per run rather than anywhere near the million-dollar target. Most tasks stalled somewhere along the way: misunderstood requirements, missing deliverables, or outputs that fell short of what the client had accepted from a real contractor.

That said, the gap is instructive rather than dismissive. These are the same models that crush competitive programming benchmarks, which suggests that real freelance work involves something the clean tests don't capture: ambiguous instructions, client-specific preferences, multi-file projects with messy dependencies, and the need to figure out what the client actually wants rather than what they literally wrote.

Why this matters

Benchmarks like this one push the field toward a more honest yardstick. A model that can ace LeetCode-style problems still has to bridge a large gap before it can replace a working professional on a real platform. RLI-style evaluation makes that gap visible in dollars — a currency everyone understands.

It also hints at where the value is for people building AI tools. The failures tend to cluster around specification understanding, long-horizon task management, and self-verification. Those are exactly the areas where better agent scaffolding, memory, and tooling could push scores upward.

Who should pay attention

If you're a developer wondering whether AI is about to eat freelance platforms, this data suggests the timeline is longer than the hype implies. If you build or invest in AI agents, it's a useful map of where current systems break. And if you're a freelancer, it's a reasonable sign that your clients still value judgment, communication, and taste — at least for now.

The million-dollar question, appropriately enough, remains open. But having a benchmark that asks it in dollar terms is a real step forward.