The harness gap

For about two months I have been bending Claude Code to work the way I work, and recently I wanted to know whether any of it was helping. So I did what you do: measure a baseline, change one thing, measure again. A lot of careful work has gone into the public benchmarks, making them fair and spreading them across the kinds of problems I deal with, so I started running my setup against those.

Within a few hours I could see, based on the structure of the testing, how the numbers are gamed by some people. Between the benchmark and the model there is a space where you can do whatever you want, nobody outside gets to look, and everything you do in there moves the score. One of my changes lifted a hallucination check by fifteen points. It will not work for anyone else, and I suspect it does not really work for me, but I have the result, and if I wanted to publish it I could. The scientific half of me wants only the changes that make the tool better. The other half noticed that the fifteen points were sitting right there.

The same fork played out at full scale while I was doing this. On 3 September the ARC Prize Foundation published its results for OpenAI's new model, GPT-6 Astra, on ARC-AGI-3. That benchmark drops an agent into games it has never seen, with no instructions, and scores how well it works out the rules and wins; every environment is solvable by a person. Under the Foundation's standard harness, with the model's reasoning effort turned all the way up, Astra scored 62.7 percent. Under a second harness built on OpenAI's own context-management settings, the same model at the same reasoning setting scored 98.6 percent, and one setting lower it scored 99.9, which became the headline. On the Sunday, Jensen Huang posted that AGI had arrived and congratulated the OpenAI team. His post does not mention a harness. It does not mention a number either, and I doubt he had the 62.7 in mind.

The fair reading of this is that nothing went wrong. ARC-AGI-3 is a serious test run by a nonprofit that runs every evaluation itself, publishes what each run cost, and refuses to show self-reported scores because, in its own words, they "often vary in dataset curation, prompting methods, and many other factors, which prevents an apples-to-apples comparison of results." The Foundation ran both harnesses and published both numbers side by side, with the conditions labelled. The second harness is two documented settings in OpenAI's API, one that keeps the model's reasoning state between calls and one that compacts a long conversation instead of dropping the start of it, and any developer with a key can switch them on. François Chollet, who created the benchmark, drew the line in July: a harness custom-made for the benchmark is out, general-purpose API settings available to every user are fine, and the parity problem between providers is acceptable "as long as the settings and the cost are clearly reported." OpenAI put the general point better than its critics have, in its own July post: benchmarks never measure just the model, they also measure the technical setup around it. Nobody buys a bare model, so the 99.9 is arguably the honest number for the product on sale.

The weights did not change between the two runs, and neither did the games or the referee, and at the same reasoning setting the scores came out thirty-six points apart. Neither number is the model's score, because there is no such thing anymore. Each is the score of a pair, a model and a harness, and the half of the pair that made the difference is the half you cannot see. The Foundation's own description of the winning harness says it preserves "opaque reasoning state between requests". The state that carried the model from 62.7 to 98.6 is hidden by design, from the referee as much as from you. The Foundation also tried six reasoning settings on each harness, which lets you compare the two levers directly. On the standard harness, the model's own reasoning dial moved the score across a range from 17.5 percent to 62.7. With the adapter on, the same six settings all landed between 96.7 and 99.9. The hidden layer moved the result further than the model's own thinking did, and once it was in place, how hard the model thought barely registered. This is also the second time it has happened this year. In July, the same two settings took OpenAI's previous model, GPT-5.6 Sol, from 7.8 percent under the official harness to 38.3 percent, which moved it from well behind Claude Opus 5 on that board to ahead of it. Both times the bigger number is the one that got repeated, and the label the referee attached to it did not get repeated with it.

On our podcast I argued the other side of this, which is that the harness is where the moat is going anyway. After a year of distillation the weights hold less of a lab's advantage than we initially thought they did, and the durable part of the product is the machinery wrapped around them: the coding harness, the context management, the two settings that carried Astra past 98. If that is where the value lives, then the harness score is the product's score, and a benchmark that strips the harness off is measuring something nobody sells. I still think that is right about buying. It is wrong about comparing. A benchmark has one job, which is to put different things under the same conditions so that a difference in the number is a difference in the things. Once the harness is allowed to vary, every score has two coordinates, and the second one is the first thing to go when the score gets repeated.

In 2011 three psychologists, Joseph Simmons, Leif Nelson and Uri Simonsohn, published a paper titled "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant". That paper is about the same second coordinate. They called the flexibility researcher degrees of freedom: which of several outcome measures to report, when to stop collecting data, which conditions to include. Each choice is defensible on its own. Made in sequence and left undisclosed, they let an honest-seeming study find almost anything, and the field spent the next decade discovering how much of its literature had been found that way. Two years later Andrew Gelman and Eric Loken showed that no fishing is even required. Their paper on the garden of forking paths describes a researcher who runs a single analysis, in good faith, and still walks a path chosen by what the data showed along the way, so the result depends on paths not taken that the researcher never saw either.

The harness is the garden of forking paths for an AI benchmark, and the space it occupies, between the weights and the score, is what I have started calling the harness gap. Keep the reasoning state between calls or throw it away. When the history fills up, compact it, or drop the oldest turns. Then the tools, the retries, the system prompt, and what counts as done. Every fork moves the score, every fork is defensible, and the person publishing gets to walk the garden first and report the path that ended highest. A position paper from May this year, "Stop Comparing LLM Agents Without Disclosing the Harness", argues that on long tasks the harness is often a stronger determinant of performance than the model it wraps, and documents cases where changing the harness reversed the ranking of the models. Its authors want mandatory harness disclosure. Simmons, Nelson and Simonsohn wanted the same thing in 2011, for the same reason. You cannot remove the flexibility, so the only remedy is to make every fork visible, and that has been the remedy since 2011.

That is also what my co-hosts heard in the 99.9 before any of us had read the harness notes. Alex used to do data science for a living, and his reaction was the one every data scientist learns early: a model that gives you 99 out of the box means you have done something wrong, and you look at the number very suspiciously until you find it. We also noticed a contradiction. The same week, OpenAI described a newer internal model as "significantly more capable than GPT-6 Astra", with "unprecedented performance in its benchmarks". Astra had just scored 99.9 on the benchmark built to measure general intelligence. There cannot be that much headroom above a saturated test of general intelligence, so one of the two sets of benchmarks is measuring something other than what it says. My guess is that a saturated benchmark is what a harness gap looks like from the outside.

The Foundation's response is the psychologists' remedy exactly. It now runs both harnesses and labels the condition next to every score, with the cost of the run alongside. The trouble is that the label lives on the leaderboard and the number lives everywhere else. It lives in a post from the chief executive of the company that sells the GPUs, and a fortnight later in the deck your vendor brings. A percentage has no slot for its conditions, so the harness falls off within a day of publication, and by the time the score reaches an operator in Brisbane it has become a fact about a model.

So the question to ask of any benchmark number that reaches you is the one the Foundation asks of itself: which harness, which settings, at what cost, run by whom. If the person quoting it cannot answer, you are holding a self-report with its degrees of freedom undisclosed. The better move, and the one I ended up making, is to stop reading other people's numbers and build your own: your work, your harness, the same harness every run, measured before and after. I wrote before about the harness being where the work lives. It is now also where the score lives, and the only harness whose forks you can see is your own. The pod's other host Jim said his measurement of Astra was more useful than the leaderboard. He gave it a job, it went away for an hour, and it came back with the work done, and he noticed the hour because he had never had one before. That is a score of a pair he actually uses and the one that actually matters.