We benchmarked a free local LLM against Claude for real work
I keep getting some version of the same question: now that a good open-weight model can run entirely on a laptop for free, why pay for AI at all? I used to answer with a shrug, because the honest answer was "I do not know, I have not tested it properly." So I stopped shrugging and ran the test.
The setup: a MacBook Pro with the M5 Max chip and 128 GB of unified memory, running gpt-oss-120b, OpenAI's open-weight 120-billion-parameter model, served locally with Apple's MLX framework. Zero dollars per token, fully offline, nothing leaves the machine. Against that, Claude Sonnet, a mid-tier paid cloud model, and Claude Haiku, a small paid cloud model. Three rounds, 10 tasks, blind judging where the answer was not obvious on its face.
The line that decided everything
I expected a spread of results across a spread of task types. What I got instead was a single, clean dividing line, and it was not the one I predicted. It was not "hard tasks versus easy tasks" or "coding versus writing." It was checkable versus uncheckable.
A checkable task is one where the output can be verified mechanically. Code where tests either pass or fail. A question with one exact correct answer. An extraction job where the field either matches the source or it does not. JSON that either validates against a schema or does not. On every one of these, the free local model tied the paid cloud models on correctness. Not "came close." Tied.
That held up even on the tasks I expected to break it. Debugging an infinite loop. A reasoning trap involving base rates, the kind of question that is deceptively easy to get wrong with confident-sounding logic. Pulling one specific fact out of a 17,000-token document, buried nowhere near the start or the end. Producing JSON constrained to a schema, which it got right 10 out of 10 times. On all four, local matched the paid models.
Where it fell over
The uncheckable tasks are the other story: strategy calls, recommendations, client-facing prose, anything where "correct" is a judgment rather than a fact. There, the local model scored clearly lower in blind judging, roughly a 6 against an 8 for the paid models. That gap by itself would just mean "use the paid model for the important stuff," which is not surprising.
What makes it worth a warning is the failure mode, not the score. The local model did not fail by going blank or hedging. It failed by confidently fabricating plausible specifics: invented multipliers, invented financial figures, competitor names that were loosely relevant at best, all delivered in the same authoritative tone as the correct answers. Nothing about the output signals "I am guessing here." You have to already know the domain to catch it, which is exactly the situation where you were hoping the model would help you.
The real cost is not quality, it is speed
If the local model ties on checkable work, what is the catch? It is operational, not qualitative. Cold starts ran 38 to 119 seconds before the first token. It processes one request at a time, so a second job queues behind the first instead of running alongside it. And it noticeably slows down on long inputs compared to the cloud models. None of that shows up in a correctness score, but all of it shows up in how the day actually feels.
The routing rule we run on now
One clean benchmark is not a law of nature, but it was clear enough that I changed how I route work. The rule is: classify the task by checkability before anything else.
| Task shape | Where it goes |
|---|---|
| Checkable, and latency does not matter | Local, free |
| Checkable, but needs speed or several jobs at once | Small paid model |
| Uncheckable judgment, or a wrong answer is expensive | Frontier paid model, never local |
There is a privacy bonus baked into that first row too. Anything touching client or personal data now defaults to local by policy, not just preference, because the data never leaves the machine in the first place.
The honest caveat
The tie on checkable work was measured with verification sitting in the loop the whole time, not as an afterthought. Local output gets checked exactly like any other AI output does: read it, run it, confirm it before anything downstream depends on it. The tie is real, but it is a tie under supervision, not a tie because the model can be trusted unsupervised. That distinction matters more than the benchmark score, and it is the same discipline I write about in why "done" from an AI is a claim, not a fact: verify first, believe second, regardless of which model produced the output.
The bottom line
So can a Mac and a free model replace paid AI? For a well-defined slice of work, yes, and that slice is bigger than most people assume: anything you can check mechanically is fair game for local, at zero cost, with nothing leaving the building. For judgment calls, strategy, and anything client-facing, no, and the risk is not that the free model looks bad, it is that it looks exactly as confident as the correct answer while being wrong. Knowing which bucket a task falls into before you run it is most of the skill.
If you are trying to work out which of your own workflows belong on which side of that line, that is exactly the kind of audit I do for clients. Send a briefand I'll help you map it out.