A GPT-4.1 Agent Passes 77.4% of Runs but Repeats Only 53.0% of Tasks
This fortnight, three sellers packaged the layer that is supposed to make an AI agent reliable: a consulting framework, a rented API, a payments identity standard. The only group that actually measured reliability found a 24.4-point gap between the number an agent posts on average and the number it can be trusted to repeat.
The harness has a name, and it still doesn't have a number
Every agent runs inside something bigger than the model itself, a loop that manages context, calls tools, and decides when a task is done. The industry now has a word for it: harness.
BCG published a framework this week naming five parts of a harness, Specs, Constitution, Control Panel, Context Hub, Quality Gates, each mapped to a piece of a computer's operating system, from application manifest to permissions layer. Quality Gates, it writes, come in four types: automated gates that catch programmatic failures, evaluation gates where a critic agent checks a worker agent, human stage gates where a named individual signs off, and regulatory gates for compliance. The framework names what a harness should contain and how it should be governed, without ever attaching a number to whether a given gate holds up when the same task runs twice.
That absence matters because a Quality Gate is, at bottom, a claim about consistency: this step will be checked correctly, this output will be verified. Naming the gate is not the same act as measuring what it is meant to prevent.
Three ways to buy the layer that's supposed to make it reliable
BCG recommends building a harness in house, tailored to a company's own processes: "build, don't buy," because context, quality signals and institutional knowledge cannot be purchased off the shelf. That is one answer to who ends up owning the harness. OpenAI, days earlier, gave the opposite one.
OpenAI's Agents API, opened to public beta on September 10th, hands developers the same harness that runs Codex and ChatGPT for Work, maintained by OpenAI rather than by the developer. The developer only chooses where the work runs: an OpenAI-managed sandbox, its own infrastructure, or one of nine named partner sandboxes, and pays only for tokens and tool use. Each new model release brings harness upgrades the developer never has to build.
That choice, build in house or rent from the lab, is what I'd tell a friend to weigh before letting OpenAI run the harness for them through the new API. Renting is a good call if the goal is to get to work quickly without spending time optimizing the task too much. But if the team already knows what it's doing, it should be thinking about customizing parts of that harness itself, starting with memory.
A third answer to who owns the harness comes from outside software entirely. Visa, Mastercard and Ant International announced on September 10th a joint Know Your Agent initiative, to let card networks, wallets and marketplaces verify an AI shopping agent's identity before it completes a purchase. Each already runs its own protocol, Visa's Trusted Agent Protocol, Mastercard's Verifiable Intent, Ant International's Agentic Mobile Protocol, and the three have so far only agreed, per their joint release, to explore common principles. Know Your Agent is not yet an interoperable standard; it is three incumbents converging on the idea that an agent, like a cardholder, needs a checkable identity. The release itself projects, in the companies' own words, that AI agents will orchestrate 3 trillion to 5 trillion dollars of global consumer commerce by 2030. PYMNTS, reporting on the announcement, adds its own March PYMNTS Intelligence estimate that outdated identity controls cost businesses close to 100 billion dollars a year in fraud, false declines and lost customers.
The one number that says whether a harness holds
None of the three offers above puts a number on reliability. IBM Research did, in a Hugging Face post published September 15th that introduces what it calls a consistency gap.
Most agent benchmarks report Mean@k: run a task several times, average the pass rate. That is the number on every leaderboard, and it answers how good an agent is on average. It does not answer what a team needs to know before shipping something: will the agent still succeed if the exact same request comes in twice? For that, IBM measured Pass^k, the share of tasks solved on every single one of k repeated runs.
On AppWorld's test_normal benchmark, 168 tasks, a ReAct agent running GPT-4.1 posted a Mean@5 of 77.4 percent, a strong average, but a Pass^5 of only 53.0 percent. Nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes cannot, nothing about the task changing between runs; it ran at temperature 0.0, so the swing was not sampling noise. IBM calls the 24.4-point difference the consistency gap, and says it is orthogonal to capability, a stronger model raises the average without necessarily closing the gap. That is IBM's own framing, not a result tested on a frontier model, and the measurement itself covers one team, one benchmark, one model.
IBM did not stop at diagnosing the gap. Its Consistency Analyzer resamples the decision points inside one recorded run to find the ones close to a coin flip, then turns the risky ones into reusable guidelines. Applying them cut the aggregate gap roughly in half, from 24.4 to 12.0 points, Pass^5 rising to 69.0 percent, Mean@5 to 81.0. On a weaker model, gpt-oss-120b, the same fix raised Pass^5 from 10.1 to 16.1 percent on the same task. The gap narrows; by IBM's own numbers, 12.0 points of it remain.
I once watched a version of this gap play out with no benchmark attached to it at all. We were tuning a scraper's search parameters, trying to land on the keyword combinations that returned the best results, and had settled on a clear list to work from. After a while the agent started mixing up the keywords, then mixing up languages entirely. I only caught it because I was checking the logs myself, by eye.
What the gap costs the people running the harness
BairesDev's Q3 2026 Dev Barometer, surveying 705 developers and 41 enterprise CTOs across more than 60 countries, found that 42 percent of developers now say AI writes at least half their code, up from just 12 percent in Q3 2025. The 705 are not simply employed developers: most, BairesDev says, are applicants in its own recruiting screen. The developer figures here come from that sample; the 41 CTOs are a separate, much smaller group.
Time saved writing code has not turned into free capacity. Sixty-seven percent spend more time reviewing AI-generated code than a year earlier, 52 percent more time debugging what it introduced, and only 21 percent still spend more than half their week writing new code from scratch. That reallocation is IBM's gap seen from the other side of the harness: output that looked done on the first pass is not output a team can rely on unchecked.
The volume behind these hours is not small. By mid-August, OpenAI's median researcher was running more than 600 dollars a day of coding-agent inference at API prices, its 90th-percentile user above 7,000. Its research organization now runs 3.1 agent-workdays of effort per human workday. 1Password, running Codex across its own delivery pipeline, reports a productivity gain of almost 21 percent for its core user cohort and a median pull request cycle time cut by nearly 11 percent. Neither figure is independently audited: the 1Password case study is published by OpenAI, not by 1Password, and the OpenAI numbers above describe OpenAI's own organization. Both point the same way: the agent work behind the hours above is already large, and growing.
That second look is also what I've settled into as a working answer, not a benchmark score. I use AI to review the code; I don't have time to review everything an agent writes. The best I can do is set up the working environment to keep alignment between my vision and what the AI builds: asking a lot of design questions, and asking the AI to ask me the maximum number of questions.
What it means
BCG names Quality Gates as one of five harness components and attaches no number to them. IBM names one gate, consistency, and attaches a number: 24.4 points, cut to 12.0 with the right guidelines, on one benchmark, one model, one team's measurement. That asymmetry is this fortnight's story: vendors selling a harness, bespoke framework, rented API, identity standard, are selling components. The team measuring whether one works is a research group publishing a diagnostic tool on a blog.
BCG's own client results in that framework, advisors tripling client time, a five-times efficiency gain, design-to-deployment speed 50 percent higher, are its self-reported account of its own engagements, not an independently measured baseline. They belong as a claim, not as evidence a harness works.
That gap between what gets named and what gets measured is also where I'd push back on BCG's own list. Their five components matched relatively well onto the eight levers we use in our own agentic AI methodology, and they clearly have a unified way of putting them together. But a harness depends on the usage and situation it gets deployed into, and I don't think it should be standardized the way their framework presents it. What they've built looks less like a fixed harness and more like a decision matrix, which is closer to what we've built too.
Buying a harness, this fortnight, means five named boxes, a rented loop, or a payments identity three companies have only agreed to explore. Measuring one means resampling five runs of the same task and reporting the number that does not flatter the average. Until sellers publish that second number, a harness with Quality Gates in its brochure and a harness actually checked for consistency are, from the outside, indistinguishable.
Comments